Nov 03 20:00:51.843950 ip-10-0-140-102 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 20:00:51.843962 ip-10-0-140-102 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 20:00:51.843971 ip-10-0-140-102 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 20:00:51.844304 ip-10-0-140-102 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 20:01:02.093753 ip-10-0-140-102 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 20:01:02.093777 ip-10-0-140-102 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a7194bf74d8143ca9bc0747d953ab879 -- Nov 03 20:02:57.640339 ip-10-0-140-102 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:02:58.027387 ip-10-0-140-102 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:02:58.027387 ip-10-0-140-102 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:02:58.027387 ip-10-0-140-102 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:02:58.027387 ip-10-0-140-102 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:02:58.030645 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030574 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.030645 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030645 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030650 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030654 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030657 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030663 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030666 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030669 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030675 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030678 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030681 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030683 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030686 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030689 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030691 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030694 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030696 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030701 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030705 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.030720 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030708 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030712 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030714 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030718 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030720 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030724 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030726 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030730 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030733 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030736 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030738 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030741 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030744 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030746 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030749 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030752 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030755 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030757 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030760 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030763 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.031174 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030765 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030768 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030771 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030773 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030776 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030779 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030782 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030784 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030787 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030790 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030792 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030795 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030797 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030800 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030803 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030806 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030809 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030811 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030814 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.031954 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030817 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030819 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030824 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030828 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030832 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030835 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030838 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030841 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030843 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030846 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030848 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030851 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030854 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030856 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030859 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030862 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030864 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030867 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030869 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.032480 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030872 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030874 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030878 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030880 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030883 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030886 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030888 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030891 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030894 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030897 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030902 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030905 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030908 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030910 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030912 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030915 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030917 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030920 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030923 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.032967 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030925 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030928 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030930 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030933 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030935 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030938 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030940 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.030944 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033449 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033461 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033465 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033469 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033472 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033475 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.033472 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033479 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033482 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033485 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033488 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033490 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033493 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033495 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033498 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033500 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033503 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033505 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033508 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033511 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033513 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033516 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033518 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033521 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033523 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033526 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.033803 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033529 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033532 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033534 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033537 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033539 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033542 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033545 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033547 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033550 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033553 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033555 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033558 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033560 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033563 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033566 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033568 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033571 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033573 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033576 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033578 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.034294 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033581 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033583 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033586 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033588 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033591 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033593 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033596 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033599 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033602 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033604 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033607 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033610 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033612 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033615 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033620 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033624 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033628 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033631 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033634 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.034789 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033636 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033639 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033642 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033645 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033648 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033650 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033653 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033657 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033659 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033662 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033664 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033667 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033669 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033672 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033674 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033677 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033680 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033682 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033685 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.035275 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033691 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033694 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033697 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033699 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033702 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033704 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033707 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033709 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033712 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033715 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033717 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033720 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033722 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033725 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033730 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033733 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033737 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033740 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033743 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.035734 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033745 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.033748 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034252 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034258 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034261 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034264 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034267 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034269 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034272 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034275 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034277 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034281 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034283 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034286 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034289 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034292 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034294 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034297 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034301 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.036251 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034305 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034309 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034312 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034315 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034317 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034320 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034323 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034326 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034328 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034331 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034333 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034336 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034338 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034341 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034344 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034349 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034351 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034354 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034356 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.036701 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034359 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034361 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034364 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034366 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034368 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034371 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034374 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034377 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034379 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034382 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034384 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034386 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034389 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034391 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034394 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034396 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034399 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034401 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034404 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034406 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.037209 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034409 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034411 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034414 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034417 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034419 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034422 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034425 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034427 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034430 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034432 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034435 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034438 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034441 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034443 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034445 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034448 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034450 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034453 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034456 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034458 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.037727 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034460 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034463 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034467 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034470 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034473 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034475 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034478 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034480 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034483 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034486 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034489 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034491 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034493 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034496 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034499 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034501 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034503 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034507 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034511 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.038215 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034513 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034516 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034521 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034523 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034526 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034528 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034531 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034533 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.034535 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034630 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034637 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034645 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034650 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034654 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034658 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034662 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034666 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034669 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034672 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034676 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:02:58.038715 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034679 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034682 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034686 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034688 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034691 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034695 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034697 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034701 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034704 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034709 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034713 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034716 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034720 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034723 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034726 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034729 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034732 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034735 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034738 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034741 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034744 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034749 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034753 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034755 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034759 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:02:58.039250 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034762 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034765 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034769 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034773 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034776 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034779 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034782 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034787 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034792 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034795 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034798 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034801 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034804 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034807 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034810 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034813 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034817 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034820 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034824 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034827 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034830 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034834 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034837 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034840 2578 flags.go:64] FLAG: --help="false" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034844 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.039859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034847 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034850 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034853 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034856 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034859 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034862 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034865 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034874 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034877 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034881 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034884 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034887 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034889 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034892 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034895 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034899 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034903 2578 flags.go:64] FLAG: --lock-file="" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034906 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034909 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034912 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034917 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034920 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034923 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:02:58.040495 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034928 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034931 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034934 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034937 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034940 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034944 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034947 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034951 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034954 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034957 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034961 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034964 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034966 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034969 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034972 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034979 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034982 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034986 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034989 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034992 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034995 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.034998 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035001 2578 flags.go:64] FLAG: --port="10250" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035003 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035008 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e6fe418f2be395f6" Nov 03 20:02:58.041171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035011 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035014 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035017 2578 flags.go:64] FLAG: --register-node="true" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035020 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035023 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035026 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035030 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035034 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035038 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035041 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035044 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035047 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035050 2578 flags.go:64] FLAG: --runonce="false" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035053 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035056 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035059 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035062 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035065 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035085 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035090 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035095 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035098 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035101 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035104 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035106 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035110 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:02:58.041780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035113 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035116 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035121 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035124 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035127 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035136 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035139 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035142 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035145 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035148 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035151 2578 flags.go:64] FLAG: --v="2" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035156 2578 flags.go:64] FLAG: --version="false" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035160 2578 flags.go:64] FLAG: --vmodule="" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035165 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035168 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035304 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035308 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035311 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035314 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035317 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035320 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.042451 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035325 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035327 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035330 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035333 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035336 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035338 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035341 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035343 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035346 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035349 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035352 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035355 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035357 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035360 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035362 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035368 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035372 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035374 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035377 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.042983 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035380 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035382 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035386 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035388 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035392 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035395 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035397 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035400 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035402 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035405 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035408 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035414 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035418 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035421 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035424 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035427 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035429 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035432 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035434 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.043457 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035437 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035439 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035442 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035445 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035447 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035450 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035452 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035455 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035459 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035461 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035464 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035466 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035469 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035472 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035474 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035477 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035480 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035483 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035486 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035488 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.043938 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035491 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035493 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035496 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035499 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035501 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035503 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035506 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035508 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035511 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035513 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035516 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035521 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035523 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035526 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035528 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035530 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035533 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035536 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.044436 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035538 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035542 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035544 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035547 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035550 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035552 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035555 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035557 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035560 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035563 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035566 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035568 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035571 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035574 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035576 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035579 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035581 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035584 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035587 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035589 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.044890 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035592 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.035619 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035661 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035665 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035668 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035670 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035673 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035676 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035678 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035681 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035683 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035686 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035689 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035691 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035695 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035698 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.045374 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035701 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035703 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035705 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035708 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035711 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035714 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035717 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035720 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035723 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035725 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035728 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035730 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035733 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035735 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035738 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035740 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035743 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035746 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035748 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035751 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.045753 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035754 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035757 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035759 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035762 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035764 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035767 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035770 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035772 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035775 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035777 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035780 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035783 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035786 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035788 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035791 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035793 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035796 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035799 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035802 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.046290 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035804 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035807 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035809 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035812 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035815 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035819 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035822 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035825 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035828 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035830 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035833 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035835 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035838 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035840 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035843 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035845 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035848 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035850 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035853 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.046739 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035855 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035860 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035863 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035867 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035869 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035872 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035875 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035878 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035880 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035883 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035885 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035888 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035890 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035893 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035895 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035898 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035900 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035903 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035905 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035908 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.047216 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035910 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035913 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035915 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035918 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035920 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035922 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035925 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035929 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035932 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035934 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035936 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035939 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035942 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.035944 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.042749 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.042764 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.042780 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.042785 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042855 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.047690 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042860 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042865 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042869 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042873 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042876 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042879 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042882 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042885 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042888 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042890 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042893 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042896 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042898 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042901 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042903 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042906 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042908 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042911 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042913 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.048183 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042916 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042918 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042921 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042924 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042926 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042929 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042931 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042934 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042936 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042940 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042943 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042946 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042949 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042951 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042954 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042957 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042960 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042962 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042965 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042968 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.048631 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042970 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042973 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042975 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042978 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042980 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042983 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042988 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042991 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042994 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.042997 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043000 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043002 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043005 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043008 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043010 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043013 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043016 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043019 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043021 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.049129 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043024 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043027 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043031 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043035 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043039 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043042 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043045 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043048 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043050 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043053 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043055 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043058 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043061 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043064 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043066 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043081 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043084 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043086 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043089 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043092 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.049613 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043094 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043097 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043099 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043102 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043105 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043107 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043110 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043112 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043115 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043118 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043120 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043123 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043125 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043129 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043132 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043135 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043138 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043140 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043143 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.050194 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043146 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043148 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043150 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043153 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043155 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043160 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.043189 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043406 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043411 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043415 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043418 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043421 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043424 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043426 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043429 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.050648 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043431 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043434 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043437 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043439 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043442 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043444 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043447 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043449 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043452 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043455 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043458 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043461 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043464 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043473 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043476 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043478 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043481 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043483 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043486 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.051028 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043488 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043491 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043493 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043496 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043498 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043501 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043503 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043506 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043509 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043511 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043515 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043519 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043522 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043525 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043528 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043531 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043535 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043539 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.051532 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043542 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043545 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043547 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043550 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043552 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043555 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043558 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043561 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043563 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043572 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043575 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043580 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043582 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043585 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043588 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043590 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043593 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043595 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043598 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.051946 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043600 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043603 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043605 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043608 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043611 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043613 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043616 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043618 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043621 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043623 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043626 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043628 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043631 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043633 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043636 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043638 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043641 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043644 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043647 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043649 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.052426 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043652 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043654 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043664 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043667 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043669 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043672 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043674 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043677 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043679 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043682 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043684 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043687 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043690 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043692 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043694 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043697 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043700 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043702 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043705 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043707 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.052899 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043710 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.043733 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043775 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043779 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043782 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043785 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043788 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043791 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043793 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043796 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043799 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043802 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043804 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043807 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043809 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043818 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:02:58.053394 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043821 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043823 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043831 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043834 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043836 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043839 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043841 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043844 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043847 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043849 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043851 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043854 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043857 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043859 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043861 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043864 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043866 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043869 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043871 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:02:58.053814 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043874 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043876 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043880 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043883 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043886 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043889 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043896 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043898 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043902 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043905 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043908 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043911 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043913 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043921 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043924 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043927 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043929 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043932 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043935 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:02:58.054602 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043937 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043940 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043943 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043946 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043949 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043951 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043954 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043957 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043960 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043962 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043965 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043968 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043970 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043973 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043975 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043978 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043981 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043983 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043986 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043991 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:02:58.055090 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043994 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043997 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.043999 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044002 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044005 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044007 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044010 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044018 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044021 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044024 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044027 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044029 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044032 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044034 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044037 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044040 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044042 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044045 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044047 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:02:58.055557 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044050 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044052 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044055 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044057 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044060 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044062 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044065 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044078 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044081 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044083 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044086 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044088 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044094 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044097 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:58.044100 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.044178 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.047112 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.047913 2578 server.go:1014] "Starting client certificate rotation" Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.048031 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:02:58.056002 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.048064 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:02:58.056550 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.049762 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:02:58.056550 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.051609 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:02:58.065241 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.065224 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:02:58.067962 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.067948 2578 log.go:25] "Validated CRI v1 image API" Nov 03 20:02:58.069474 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.069350 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:02:58.071583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.071560 2578 fs.go:135] Filesystem UUIDs: map[200ac875-8a5f-406f-a9c6-c9f934e72ba2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89b1c9fd-f13d-4de3-b435-c947ba25d5e1:/dev/nvme0n1p3] Nov 03 20:02:58.071668 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.071581 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:02:58.078172 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.078040 2578 manager.go:217] Machine: {Timestamp:2026-11-03 20:02:58.076688158 +0000 UTC m=+0.317448633 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097715 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279b67898c945b792fe3c398371621 SystemUUID:ec279b67-898c-945b-792f-e3c398371621 BootID:a7194bf7-4d81-43ca-9bc0-747d953ab879 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:f3:60:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:f3:60:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:90:41:c9:4e:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:02:58.078172 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.078162 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:02:58.078341 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.078258 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:02:58.081440 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.081419 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:02:58.081571 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.081441 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:02:58.081613 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.081572 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:02:58.081613 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.081581 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:02:58.081613 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.081610 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:02:58.081691 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.081627 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:02:58.082516 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.082506 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:02:58.085543 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.085532 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:02:58.085666 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.085657 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:02:58.086361 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.086351 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:02:58.086391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.086365 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:02:58.086391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.086374 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:02:58.086391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.086383 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:02:58.086391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.086390 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:02:58.086717 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.086706 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:02:58.089744 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.089727 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:02:58.093958 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.093946 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:02:58.094688 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094676 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:02:58.094724 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094706 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:02:58.094889 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094874 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:02:58.094918 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094904 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:02:58.094944 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094919 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:02:58.094944 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094932 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:02:58.094992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094947 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:02:58.094992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.094956 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:02:58.095450 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.095441 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:02:58.095484 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.095451 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:02:58.095484 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.095458 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:02:58.095484 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.095464 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:02:58.095484 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.095471 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:02:58.095484 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.095479 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:02:58.096105 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.096095 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:02:58.096140 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.096107 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:02:58.099527 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.099514 2578 server.go:1263] "Started kubelet" Nov 03 20:02:58.099664 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.099618 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:02:58.099750 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.099610 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:02:58.099798 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.099773 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:02:58.100215 ip-10-0-140-102 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:02:58.100767 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.100751 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:02:58.106210 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.105718 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:02:58.106563 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.106546 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:02:58.108125 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.108110 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:02:58.108533 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.108518 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:02:58.108622 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.108601 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.109053 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109034 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:02:58.109053 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109035 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:02:58.109193 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109057 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:02:58.109193 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109174 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:02:58.109193 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109183 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:02:58.109352 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109336 2578 factory.go:55] Registering systemd factory Nov 03 20:02:58.109391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109356 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:02:58.109562 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109542 2578 factory.go:153] Registering CRI-O factory Nov 03 20:02:58.109562 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109556 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 20:02:58.109643 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109622 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:02:58.109643 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109636 2578 factory.go:103] Registering Raw factory Nov 03 20:02:58.109700 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.109645 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 20:02:58.110441 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.110425 2578 manager.go:319] Starting recovery of all containers Nov 03 20:02:58.121852 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.121769 2578 manager.go:324] Recovery completed Nov 03 20:02:58.126552 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.126539 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.128311 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.128292 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.128391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.128325 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.128391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.128339 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.129207 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.129165 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:02:58.129207 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.129181 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:02:58.129312 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.129279 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:02:58.129312 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.129291 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:02:58.129312 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.129309 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:02:58.129462 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.129361 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.130168 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.130153 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.130224 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.130186 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.130224 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.130201 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.130356 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.129166 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-102.ec2.internal.18e4a18ce462ffdb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-102.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-102.ec2.internal,},FirstTimestamp:2026-11-03 20:02:58.099486683 +0000 UTC m=+0.340247148,LastTimestamp:2026-11-03 20:02:58.099486683 +0000 UTC m=+0.340247148,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-102.ec2.internal,}" Nov 03 20:02:58.134880 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.134862 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:02:58.147040 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.147016 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 20:02:58.147208 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.147176 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 20:02:58.147596 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.147455 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-102.ec2.internal.18e4a18ce61ad049 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-102.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-102.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-102.ec2.internal,},FirstTimestamp:2026-11-03 20:02:58.128310345 +0000 UTC m=+0.369070820,LastTimestamp:2026-11-03 20:02:58.128310345 +0000 UTC m=+0.369070820,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-102.ec2.internal,}" Nov 03 20:02:58.147760 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.147737 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 20:02:58.156699 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.156674 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5r5q" Nov 03 20:02:58.163733 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.163713 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5r5q" Nov 03 20:02:58.175204 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.175137 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-102.ec2.internal.18e4a18ce61b23ee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-102.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-102.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-102.ec2.internal,},FirstTimestamp:2026-11-03 20:02:58.128331758 +0000 UTC m=+0.369092224,LastTimestamp:2026-11-03 20:02:58.128331758 +0000 UTC m=+0.369092224,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-102.ec2.internal,}" Nov 03 20:02:58.209494 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.209479 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.256955 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.256940 2578 policy_none.go:50] "Start" Nov 03 20:02:58.257027 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.256966 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:02:58.257027 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.256981 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:02:58.258210 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.258198 2578 policy_none.go:44] "Start" Nov 03 20:02:58.290646 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.290594 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:02:58.292806 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.292793 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 20:02:58.292891 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.292855 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:02:58.292891 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.292867 2578 server.go:87] "Starting device plugin registration server" Nov 03 20:02:58.293126 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.293114 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:02:58.293174 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.293130 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:02:58.293220 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.293203 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:02:58.293312 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.293295 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:02:58.293312 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.293309 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:02:58.293782 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.293761 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:02:58.293872 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.293815 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.347281 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.347250 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:02:58.348388 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.348374 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:02:58.348445 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.348395 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:02:58.348445 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.348413 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:02:58.348549 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.348449 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:02:58.360511 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.360490 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-102.ec2.internal\" not found" node="ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.363850 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.363837 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:02:58.393946 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.393920 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.394720 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.394706 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.394805 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.394735 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.394805 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.394750 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.394805 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.394780 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.416641 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.416622 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.416641 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.416641 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-102.ec2.internal\": node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.449288 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.449268 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal"] Nov 03 20:02:58.449370 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.449331 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.450027 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.450011 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.450123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.450042 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.450123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.450056 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.451301 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451287 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.451457 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.451503 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451471 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.451906 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451890 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.451987 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451911 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.451987 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451935 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.451987 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451950 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.451987 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451918 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.452140 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.451993 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.453122 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.453108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.453193 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.453132 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:02:58.453752 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.453736 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:02:58.453812 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.453758 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:02:58.453812 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.453772 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:02:58.460258 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.460243 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.470402 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.470384 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-102.ec2.internal\" not found" node="ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.473159 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.473143 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-102.ec2.internal\" not found" node="ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.510599 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.510579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.510655 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.510616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.510655 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.510630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/880c789d3d697e88c74ab8f8cf26d9d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" (UID: \"880c789d3d697e88c74ab8f8cf26d9d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.561225 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.561051 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.611153 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.611135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.611215 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.611159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.611215 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.611175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/880c789d3d697e88c74ab8f8cf26d9d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" (UID: \"880c789d3d697e88c74ab8f8cf26d9d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.611289 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.611216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/880c789d3d697e88c74ab8f8cf26d9d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" (UID: \"880c789d3d697e88c74ab8f8cf26d9d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.611289 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.611229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.611289 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.611241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.661959 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.661940 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.762362 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.762343 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.773506 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.773486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.775995 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:58.775976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:02:58.862638 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.862591 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:58.963000 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:58.962985 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:02:59.034779 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.034760 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:02:59.047818 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.047796 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:02:59.047933 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.047908 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:02:59.047989 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.047947 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:02:59.087051 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.087033 2578 apiserver.go:52] "Watching apiserver" Nov 03 20:02:59.109178 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.109160 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:02:59.109260 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.109163 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:02:59.167045 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.167020 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:02:59.167245 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.167136 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:57:58 +0000 UTC" deadline="2028-06-02 06:54:18.168271958 +0000 UTC" Nov 03 20:02:59.167245 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.167174 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13834h51m19.001101587s" Nov 03 20:02:59.167347 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.167326 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-947nk","openshift-cluster-node-tuning-operator/tuned-qptp7","openshift-image-registry/node-ca-bkb5w","openshift-multus/network-metrics-daemon-8vwxs","openshift-ovn-kubernetes/ovnkube-node-2chqk","openshift-dns/node-resolver-zmtdq","openshift-network-diagnostics/network-check-target-7zg6w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk","kube-system/global-pull-secret-syncer-kxtqd","kube-system/konnectivity-agent-2bsd7","openshift-multus/multus-additional-cni-plugins-d45vs","openshift-multus/multus-dk9sk"] Nov 03 20:02:59.169142 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.169120 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.169997 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.169977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.172327 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.172312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.173507 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.173488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.173602 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.173563 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:02:59.174790 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.174774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.177687 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.177658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.178913 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.178675 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:02:59.178913 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.178761 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:02:59.179685 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.179668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:02:59.179761 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.179720 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:02:59.180967 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.180950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.182770 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.182748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.182880 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.182799 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:02:59.184451 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.184434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.185698 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.185679 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4f2k\"" Nov 03 20:02:59.185885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.185867 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.185950 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.185888 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:02:59.185950 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.185906 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.185950 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.185928 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.186155 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.186142 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.186334 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.186318 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.186492 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.186479 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hwv2m\"" Nov 03 20:02:59.187947 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.187925 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal"] Nov 03 20:02:59.188026 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.188016 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194389 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194413 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194467 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqzwq\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194467 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194467 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194528 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194533 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.194573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.194550 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:02:59.195256 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.195242 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-phn2h\"" Nov 03 20:02:59.199924 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.199904 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.202424 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.202407 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:02:59.202501 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.202467 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c278j" Nov 03 20:02:59.205514 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.205499 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:02:59.205654 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.205640 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal"] Nov 03 20:02:59.210373 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.210356 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:02:59.213464 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:02:59.213464 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-tmp\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.213614 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vs5k\" (UniqueName: \"kubernetes.io/projected/331068c4-2a21-4ab6-9fbf-ea1ce605d767-kube-api-access-7vs5k\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.213614 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-cnibin\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.213614 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.213614 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-multus-daemon-config\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.213814 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-conf\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.213814 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mk2\" (UniqueName: \"kubernetes.io/projected/781d2937-5b44-4786-8c19-7629f4548399-kube-api-access-79mk2\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.213814 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.213814 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-system-cni-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.213814 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fff6f118-0ff9-4478-b759-04e2d3f63420-host\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.213814 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovn-node-metrics-cert\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213820 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8g9\" (UniqueName: \"kubernetes.io/projected/26f26e13-d1c9-4747-914f-af2fb15d7ed7-kube-api-access-4p8g9\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrt46\" (UniqueName: \"kubernetes.io/projected/edd592ba-dbd2-4026-906b-27478b901db5-kube-api-access-jrt46\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-kubernetes\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.213997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-device-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bd39722-fd14-42d7-af96-2b37b2bba425-iptables-alerter-script\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-cnibin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-kubelet\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-systemd\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.214123 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-var-lib-kubelet\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-sys-fs\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-netns\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-binary-copy\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-bin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-run\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-lib-modules\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fff6f118-0ff9-4478-b759-04e2d3f63420-serviceca\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4ms\" (UniqueName: \"kubernetes.io/projected/fff6f118-0ff9-4478-b759-04e2d3f63420-kube-api-access-cv4ms\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h556\" (UniqueName: \"kubernetes.io/projected/2b318e41-0c85-4303-addc-336f59f4a4e7-kube-api-access-7h556\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f26e13-d1c9-4747-914f-af2fb15d7ed7-tmp-dir\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-socket-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.214583 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214573 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-node-log\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-system-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bd39722-fd14-42d7-af96-2b37b2bba425-host-slash\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8e9079-e494-4815-a2dc-fc3aa2763628-agent-certs\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-multus\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214868 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f26e13-d1c9-4747-914f-af2fb15d7ed7-hosts-file\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-kubelet-config\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-etc-kubernetes\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-kubelet\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-config\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.214990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-script-lib\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-conf-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzbb\" (UniqueName: \"kubernetes.io/projected/8ecf92d9-64ba-4348-9124-5f02d8032c9c-kube-api-access-9pzbb\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mng2w\" (UniqueName: \"kubernetes.io/projected/2bd39722-fd14-42d7-af96-2b37b2bba425-kube-api-access-mng2w\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-systemd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-etc-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8e9079-e494-4815-a2dc-fc3aa2763628-konnectivity-ca\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-socket-dir-parent\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-k8s-cni-cncf-io\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysconfig\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-netd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-registration-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-systemd-units\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-slash\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-var-lib-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.215992 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-bin\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-env-overrides\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-hostroot\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-multus-certs\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-sys\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-ovn\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-log-socket\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-cni-binary-copy\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-host\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-tuned\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-dbus\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.215979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-netns\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.216027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-modprobe-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.216686 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.216096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7bf\" (UniqueName: \"kubernetes.io/projected/31637ee3-ae67-4836-992c-3553f96a85e5-kube-api-access-tl7bf\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.234999 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.234976 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880c789d3d697e88c74ab8f8cf26d9d3.slice/crio-ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0 WatchSource:0}: Error finding container ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0: Status 404 returned error can't find the container with id ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0 Nov 03 20:02:59.235201 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.235177 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59661663947a8edd85fd5fa4c740ff44.slice/crio-fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866 WatchSource:0}: Error finding container fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866: Status 404 returned error can't find the container with id fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866 Nov 03 20:02:59.241971 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.241951 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:02:59.295735 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.295714 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:02:59.296053 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296038 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kzs97\"" Nov 03 20:02:59.296116 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296089 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.296160 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296154 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfgm6\"" Nov 03 20:02:59.296450 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.296450 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296418 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.296450 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296431 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:02:59.296450 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296440 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.296450 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296428 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296483 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b4qhs\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296484 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296487 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296648 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296675 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kzdq\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296662 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296686 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qw54\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296685 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:02:59.296825 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.296770 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:02:59.317420 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-node-log\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.317482 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-system-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.317482 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.317482 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.317482 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bd39722-fd14-42d7-af96-2b37b2bba425-host-slash\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8e9079-e494-4815-a2dc-fc3aa2763628-agent-certs\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-node-log\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-system-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317592 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317599 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bd39722-fd14-42d7-af96-2b37b2bba425-host-slash\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-multus\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-multus\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.317652 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f26e13-d1c9-4747-914f-af2fb15d7ed7-hosts-file\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f26e13-d1c9-4747-914f-af2fb15d7ed7-hosts-file\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-kubelet-config\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-etc-kubernetes\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-kubelet\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-kubelet-config\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-config\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317802 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-etc-kubernetes\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-script-lib\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317820 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-kubelet\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-conf-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzbb\" (UniqueName: \"kubernetes.io/projected/8ecf92d9-64ba-4348-9124-5f02d8032c9c-kube-api-access-9pzbb\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-conf-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mng2w\" (UniqueName: \"kubernetes.io/projected/2bd39722-fd14-42d7-af96-2b37b2bba425-kube-api-access-mng2w\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-systemd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318191 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.317947 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-etc-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318084 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-systemd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.318107 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.318173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret podName:25244768-eb42-47ea-a28d-20609fc12f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:02:59.818150745 +0000 UTC m=+2.058911221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret") pod "global-pull-secret-syncer-kxtqd" (UID: "25244768-eb42-47ea-a28d-20609fc12f42") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-etc-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8e9079-e494-4815-a2dc-fc3aa2763628-konnectivity-ca\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-socket-dir-parent\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-k8s-cni-cncf-io\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-k8s-cni-cncf-io\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysconfig\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-netd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-config\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-socket-dir-parent\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.318908 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-registration-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysconfig\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-script-lib\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-systemd-units\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-netd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-slash\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318504 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-systemd-units\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318494 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-registration-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-slash\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-var-lib-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-bin\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-env-overrides\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-var-lib-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-hostroot\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-bin\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-multus-certs\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-multus-certs\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-sys\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.319713 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318698 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-hostroot\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-ovn\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8e9079-e494-4815-a2dc-fc3aa2763628-konnectivity-ca\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-log-socket\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-ovn\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-sys\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-log-socket\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-cni-binary-copy\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-host\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-tuned\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-dbus\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-env-overrides\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-netns\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-host\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.318998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-modprobe-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-dbus\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7bf\" (UniqueName: \"kubernetes.io/projected/31637ee3-ae67-4836-992c-3553f96a85e5-kube-api-access-tl7bf\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:02:59.320536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-modprobe-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-tmp\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-netns\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vs5k\" (UniqueName: \"kubernetes.io/projected/331068c4-2a21-4ab6-9fbf-ea1ce605d767-kube-api-access-7vs5k\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-cnibin\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-multus-daemon-config\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-conf\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319260 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-cnibin\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mk2\" (UniqueName: \"kubernetes.io/projected/781d2937-5b44-4786-8c19-7629f4548399-kube-api-access-79mk2\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-cni-binary-copy\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-system-cni-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fff6f118-0ff9-4478-b759-04e2d3f63420-host\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovn-node-metrics-cert\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.321460 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319463 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fff6f118-0ff9-4478-b759-04e2d3f63420-host\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-system-cni-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8g9\" (UniqueName: \"kubernetes.io/projected/26f26e13-d1c9-4747-914f-af2fb15d7ed7-kube-api-access-4p8g9\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.319540 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.319586 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:02:59.819568872 +0000 UTC m=+2.060329341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-conf\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt46\" (UniqueName: \"kubernetes.io/projected/edd592ba-dbd2-4026-906b-27478b901db5-kube-api-access-jrt46\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-kubernetes\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-device-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bd39722-fd14-42d7-af96-2b37b2bba425-iptables-alerter-script\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-cnibin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-kubelet\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-systemd\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-var-lib-kubelet\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-sys-fs\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.322343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-netns\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-binary-copy\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-cnibin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-bin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-kubernetes\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.319984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-run\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-device-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-lib-modules\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-multus-daemon-config\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-systemd\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-sys-fs\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-netns\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-run\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-lib-modules\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fff6f118-0ff9-4478-b759-04e2d3f63420-serviceca\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.322970 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-bin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4ms\" (UniqueName: \"kubernetes.io/projected/fff6f118-0ff9-4478-b759-04e2d3f63420-kube-api-access-cv4ms\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h556\" (UniqueName: \"kubernetes.io/projected/2b318e41-0c85-4303-addc-336f59f4a4e7-kube-api-access-7h556\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-var-lib-kubelet\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f26e13-d1c9-4747-914f-af2fb15d7ed7-tmp-dir\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-socket-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.320715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-kubelet\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.321292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f26e13-d1c9-4747-914f-af2fb15d7ed7-tmp-dir\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.321353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bd39722-fd14-42d7-af96-2b37b2bba425-iptables-alerter-script\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.321450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-binary-copy\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.321718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fff6f118-0ff9-4478-b759-04e2d3f63420-serviceca\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.321815 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-socket-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.322713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-tmp\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.322788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-tuned\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.322795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovn-node-metrics-cert\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.323498 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.323100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8e9079-e494-4815-a2dc-fc3aa2763628-agent-certs\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.352278 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.352215 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" event={"ID":"880c789d3d697e88c74ab8f8cf26d9d3","Type":"ContainerStarted","Data":"ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0"} Nov 03 20:02:59.353221 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.353195 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" event={"ID":"59661663947a8edd85fd5fa4c740ff44","Type":"ContainerStarted","Data":"fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866"} Nov 03 20:02:59.356502 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.356485 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:02:59.356589 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.356508 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:02:59.356589 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.356518 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vzpbh for pod openshift-network-diagnostics/network-check-target-7zg6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:02:59.356589 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.356566 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh podName:57170d5a-5e31-40d9-aebd-645e3c12c610 nodeName:}" failed. No retries permitted until 2026-11-03 20:02:59.856553495 +0000 UTC m=+2.097313979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vzpbh" (UniqueName: "kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh") pod "network-check-target-7zg6w" (UID: "57170d5a-5e31-40d9-aebd-645e3c12c610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:02:59.357335 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.357308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mng2w\" (UniqueName: \"kubernetes.io/projected/2bd39722-fd14-42d7-af96-2b37b2bba425-kube-api-access-mng2w\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.359285 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.359258 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7bf\" (UniqueName: \"kubernetes.io/projected/31637ee3-ae67-4836-992c-3553f96a85e5-kube-api-access-tl7bf\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.359408 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.359386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vs5k\" (UniqueName: \"kubernetes.io/projected/331068c4-2a21-4ab6-9fbf-ea1ce605d767-kube-api-access-7vs5k\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.360555 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.360527 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzbb\" (UniqueName: \"kubernetes.io/projected/8ecf92d9-64ba-4348-9124-5f02d8032c9c-kube-api-access-9pzbb\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.360636 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.360576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h556\" (UniqueName: \"kubernetes.io/projected/2b318e41-0c85-4303-addc-336f59f4a4e7-kube-api-access-7h556\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.361291 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.361275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mk2\" (UniqueName: \"kubernetes.io/projected/781d2937-5b44-4786-8c19-7629f4548399-kube-api-access-79mk2\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.361369 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.361347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4ms\" (UniqueName: \"kubernetes.io/projected/fff6f118-0ff9-4478-b759-04e2d3f63420-kube-api-access-cv4ms\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.363526 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.363504 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8g9\" (UniqueName: \"kubernetes.io/projected/26f26e13-d1c9-4747-914f-af2fb15d7ed7-kube-api-access-4p8g9\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.363742 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.363721 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt46\" (UniqueName: \"kubernetes.io/projected/edd592ba-dbd2-4026-906b-27478b901db5-kube-api-access-jrt46\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.485874 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.485816 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:02:59.497681 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.497661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:02:59.504674 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.504649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:02:59.507160 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.507128 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd39722_fd14_42d7_af96_2b37b2bba425.slice/crio-66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b WatchSource:0}: Error finding container 66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b: Status 404 returned error can't find the container with id 66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b Nov 03 20:02:59.512059 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.512036 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecf92d9_64ba_4348_9124_5f02d8032c9c.slice/crio-849f930059b85758427e177f2659d2ffd0c83fa136afe4fef2035556fb73db46 WatchSource:0}: Error finding container 849f930059b85758427e177f2659d2ffd0c83fa136afe4fef2035556fb73db46: Status 404 returned error can't find the container with id 849f930059b85758427e177f2659d2ffd0c83fa136afe4fef2035556fb73db46 Nov 03 20:02:59.525612 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.525593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:02:59.532999 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.532979 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff6f118_0ff9_4478_b759_04e2d3f63420.slice/crio-eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660 WatchSource:0}: Error finding container eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660: Status 404 returned error can't find the container with id eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660 Nov 03 20:02:59.545382 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.545359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:02:59.546434 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.546416 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:02:59.552057 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.551968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:02:59.554230 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.554211 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331068c4_2a21_4ab6_9fbf_ea1ce605d767.slice/crio-07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39 WatchSource:0}: Error finding container 07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39: Status 404 returned error can't find the container with id 07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39 Nov 03 20:02:59.557291 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.557267 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f26e13_d1c9_4747_914f_af2fb15d7ed7.slice/crio-7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f WatchSource:0}: Error finding container 7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f: Status 404 returned error can't find the container with id 7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f Nov 03 20:02:59.560045 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.560024 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31637ee3_ae67_4836_992c_3553f96a85e5.slice/crio-f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3 WatchSource:0}: Error finding container f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3: Status 404 returned error can't find the container with id f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3 Nov 03 20:02:59.565787 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.565764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:02:59.574065 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.574041 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8e9079_e494_4815_a2dc_fc3aa2763628.slice/crio-eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8 WatchSource:0}: Error finding container eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8: Status 404 returned error can't find the container with id eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8 Nov 03 20:02:59.578398 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.578382 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:02:59.579511 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.579486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dk9sk" Nov 03 20:02:59.587651 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.587626 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd592ba_dbd2_4026_906b_27478b901db5.slice/crio-954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe WatchSource:0}: Error finding container 954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe: Status 404 returned error can't find the container with id 954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe Nov 03 20:02:59.590420 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:02:59.590357 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781d2937_5b44_4786_8c19_7629f4548399.slice/crio-14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8 WatchSource:0}: Error finding container 14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8: Status 404 returned error can't find the container with id 14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8 Nov 03 20:02:59.720616 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.720588 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:02:59.824935 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.824858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:02:59.824935 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.824918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:02:59.825128 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.825025 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:02:59.825128 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.825034 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:02:59.825128 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.825106 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:00.825069961 +0000 UTC m=+3.065830431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:02:59.825128 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.825125 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret podName:25244768-eb42-47ea-a28d-20609fc12f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:00.825117651 +0000 UTC m=+3.065878104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret") pod "global-pull-secret-syncer-kxtqd" (UID: "25244768-eb42-47ea-a28d-20609fc12f42") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:02:59.926190 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:02:59.926154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:02:59.926340 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.926318 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:02:59.926388 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.926347 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:02:59.926388 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.926364 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vzpbh for pod openshift-network-diagnostics/network-check-target-7zg6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:02:59.926458 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:02:59.926421 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh podName:57170d5a-5e31-40d9-aebd-645e3c12c610 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:00.926401968 +0000 UTC m=+3.167162430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vzpbh" (UniqueName: "kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh") pod "network-check-target-7zg6w" (UID: "57170d5a-5e31-40d9-aebd-645e3c12c610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:00.204538 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.204420 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:57:59 +0000 UTC" deadline="2028-07-19 08:05:42.555318032 +0000 UTC" Nov 03 20:03:00.204538 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.204488 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14964h2m42.35083377s" Nov 03 20:03:00.325887 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.325860 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:03:00.351167 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.351126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:00.351319 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.351284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:00.351720 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.351700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:00.351830 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.351810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:00.363389 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.363360 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dk9sk" event={"ID":"781d2937-5b44-4786-8c19-7629f4548399","Type":"ContainerStarted","Data":"14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8"} Nov 03 20:03:00.371187 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.371158 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerStarted","Data":"954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe"} Nov 03 20:03:00.390733 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.390705 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2bsd7" event={"ID":"0a8e9079-e494-4815-a2dc-fc3aa2763628","Type":"ContainerStarted","Data":"eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8"} Nov 03 20:03:00.400098 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.400071 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" event={"ID":"31637ee3-ae67-4836-992c-3553f96a85e5","Type":"ContainerStarted","Data":"f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3"} Nov 03 20:03:00.410079 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.410050 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qptp7" event={"ID":"8ecf92d9-64ba-4348-9124-5f02d8032c9c","Type":"ContainerStarted","Data":"849f930059b85758427e177f2659d2ffd0c83fa136afe4fef2035556fb73db46"} Nov 03 20:03:00.426719 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.426686 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-947nk" event={"ID":"2bd39722-fd14-42d7-af96-2b37b2bba425","Type":"ContainerStarted","Data":"66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b"} Nov 03 20:03:00.442359 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.442333 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zmtdq" event={"ID":"26f26e13-d1c9-4747-914f-af2fb15d7ed7","Type":"ContainerStarted","Data":"7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f"} Nov 03 20:03:00.444697 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.444671 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39"} Nov 03 20:03:00.449010 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.448982 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bkb5w" event={"ID":"fff6f118-0ff9-4478-b759-04e2d3f63420","Type":"ContainerStarted","Data":"eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660"} Nov 03 20:03:00.837509 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.834570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:00.837509 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.834639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:00.837509 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.834791 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:00.837509 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.834848 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret podName:25244768-eb42-47ea-a28d-20609fc12f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:02.834830624 +0000 UTC m=+5.075591085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret") pod "global-pull-secret-syncer-kxtqd" (UID: "25244768-eb42-47ea-a28d-20609fc12f42") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:00.837509 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.835299 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:00.837509 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.835351 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:02.835336289 +0000 UTC m=+5.076096748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:00.935149 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:00.935114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:00.935373 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.935339 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:03:00.935373 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.935364 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:03:00.935489 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.935377 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vzpbh for pod openshift-network-diagnostics/network-check-target-7zg6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:00.935489 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:00.935434 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh podName:57170d5a-5e31-40d9-aebd-645e3c12c610 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:02.93541503 +0000 UTC m=+5.176175502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vzpbh" (UniqueName: "kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh") pod "network-check-target-7zg6w" (UID: "57170d5a-5e31-40d9-aebd-645e3c12c610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:01.205546 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:01.205463 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:57:59 +0000 UTC" deadline="2028-07-21 08:16:39.458484454 +0000 UTC" Nov 03 20:03:01.205546 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:01.205505 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15012h13m38.252985531s" Nov 03 20:03:01.349435 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:01.348905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:01.349435 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:01.349046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:01.825648 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:01.825568 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:03:02.352553 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:02.351636 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:02.352553 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.351778 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:02.352553 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:02.351865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:02.352553 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.351952 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:02.855782 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:02.855055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:02.855782 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:02.855154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:02.855782 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.855271 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:02.855782 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.855318 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:06.855304289 +0000 UTC m=+9.096064741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:02.855782 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.855670 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:02.855782 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.855712 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret podName:25244768-eb42-47ea-a28d-20609fc12f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:06.855701336 +0000 UTC m=+9.096461801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret") pod "global-pull-secret-syncer-kxtqd" (UID: "25244768-eb42-47ea-a28d-20609fc12f42") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:02.956016 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:02.955979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:02.956193 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.956170 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:03:02.956193 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.956190 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:03:02.956307 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.956202 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vzpbh for pod openshift-network-diagnostics/network-check-target-7zg6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:02.956307 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:02.956260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh podName:57170d5a-5e31-40d9-aebd-645e3c12c610 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:06.956243069 +0000 UTC m=+9.197003541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vzpbh" (UniqueName: "kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh") pod "network-check-target-7zg6w" (UID: "57170d5a-5e31-40d9-aebd-645e3c12c610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:03.349714 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:03.349438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:03.349855 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:03.349832 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:03.471350 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:03.471315 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" event={"ID":"880c789d3d697e88c74ab8f8cf26d9d3","Type":"ContainerStarted","Data":"6070ec032b3df948b320aaa5fcce74c667b82ff7b067067832b70584fb916db4"} Nov 03 20:03:03.480514 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:03.480486 2578 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="6aa648fe65e92ebc5cbc08a0194656931b1021e5cbb8f61f29361615109b512a" exitCode=0 Nov 03 20:03:03.480620 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:03.480533 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerDied","Data":"6aa648fe65e92ebc5cbc08a0194656931b1021e5cbb8f61f29361615109b512a"} Nov 03 20:03:03.556051 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:03.555998 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" podStartSLOduration=4.555980962 podStartE2EDuration="4.555980962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:03.511222594 +0000 UTC m=+5.751983069" watchObservedRunningTime="2026-11-03 20:03:03.555980962 +0000 UTC m=+5.796741436" Nov 03 20:03:04.360477 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:04.360206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:04.360655 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:04.360481 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:04.360655 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:04.360585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:04.360866 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:04.360831 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:04.487785 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:04.486882 2578 generic.go:356] "Generic (PLEG): container finished" podID="59661663947a8edd85fd5fa4c740ff44" containerID="d9afc4f07db171013d0ad9a79e71dd753566e5f6afc2ddc0397b92eaa5b1e275" exitCode=0 Nov 03 20:03:04.487785 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:04.486973 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" event={"ID":"59661663947a8edd85fd5fa4c740ff44","Type":"ContainerDied","Data":"d9afc4f07db171013d0ad9a79e71dd753566e5f6afc2ddc0397b92eaa5b1e275"} Nov 03 20:03:04.493012 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:04.492890 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2bsd7" event={"ID":"0a8e9079-e494-4815-a2dc-fc3aa2763628","Type":"ContainerStarted","Data":"d0585d924a338955b0a189bd83abdfe37fa9fb45d9b75d7bf990aebe62f82294"} Nov 03 20:03:04.573019 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:04.572958 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2bsd7" podStartSLOduration=2.281781586 podStartE2EDuration="6.572939702s" totalImagesPullingTime="4.291158116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:04.572160274 +0000 UTC m=+6.812920749" watchObservedRunningTime="2026-11-03 20:03:04.572939702 +0000 UTC m=+6.813700179" Nov 03 20:03:05.349422 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:05.349019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:05.349422 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:05.349154 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:05.498138 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:05.497892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bkb5w" event={"ID":"fff6f118-0ff9-4478-b759-04e2d3f63420","Type":"ContainerStarted","Data":"a2e3cea69749e6c6be5dcd7a659109eebe6577b4510914293ba7f06f3c1c96c8"} Nov 03 20:03:05.503490 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:05.503461 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" event={"ID":"59661663947a8edd85fd5fa4c740ff44","Type":"ContainerStarted","Data":"6da68763abe3c3a5eb518c5b8470a1c76db47f6711ba3de4b03695c46a94a299"} Nov 03 20:03:05.597222 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:05.597161 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bkb5w" podStartSLOduration=1.9411024540000001 podStartE2EDuration="7.597142866s" totalImagesPullingTime="5.656040412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:05.537965769 +0000 UTC m=+7.778726244" watchObservedRunningTime="2026-11-03 20:03:05.597142866 +0000 UTC m=+7.837903342" Nov 03 20:03:05.597651 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:05.597612 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" podStartSLOduration=6.59760136 podStartE2EDuration="6.59760136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:05.596985994 +0000 UTC m=+7.837746468" watchObservedRunningTime="2026-11-03 20:03:05.59760136 +0000 UTC m=+7.838361834" Nov 03 20:03:06.350334 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:06.350265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:06.350461 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.350417 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:06.350932 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:06.350908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:06.351047 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.351021 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:06.508544 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:06.508176 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" event={"ID":"31637ee3-ae67-4836-992c-3553f96a85e5","Type":"ContainerStarted","Data":"3455fdf0ed25aa18ee4f53e081bdf3a61c2f4285dfeb007241d812a358c02d34"} Nov 03 20:03:06.888207 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:06.888166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:06.888393 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:06.888235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:06.888393 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.888372 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:06.888537 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.888421 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret podName:25244768-eb42-47ea-a28d-20609fc12f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:14.888409082 +0000 UTC m=+17.129169546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret") pod "global-pull-secret-syncer-kxtqd" (UID: "25244768-eb42-47ea-a28d-20609fc12f42") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:06.888764 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.888745 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:06.888840 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.888804 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:14.888787663 +0000 UTC m=+17.129548129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:06.989365 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:06.989322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:06.989565 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.989534 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:03:06.989565 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.989559 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:03:06.989668 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.989574 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vzpbh for pod openshift-network-diagnostics/network-check-target-7zg6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:06.989668 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:06.989628 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh podName:57170d5a-5e31-40d9-aebd-645e3c12c610 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:14.989612888 +0000 UTC m=+17.230373364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vzpbh" (UniqueName: "kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh") pod "network-check-target-7zg6w" (UID: "57170d5a-5e31-40d9-aebd-645e3c12c610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:07.349479 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:07.349435 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:07.349759 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:07.349590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:07.419646 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:07.419625 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:03:07.515689 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:07.515572 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" event={"ID":"31637ee3-ae67-4836-992c-3553f96a85e5","Type":"ContainerStarted","Data":"3523cde4bd5d084c441439fb2afe2f8987f8273df544cb93f4cb4490baab1c75"} Nov 03 20:03:07.926674 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:07.926026 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:07.929355 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:07.929115 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:08.298213 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.298119 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:03:07.419644351Z","UUID":"cb4bd10f-ef97-4896-aa6f-f2c3fcf92de1","Handler":null,"Name":"","Endpoint":""} Nov 03 20:03:08.299781 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.299758 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:03:08.299875 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.299786 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:03:08.352209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.352183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:08.352339 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:08.352318 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:08.352655 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.352634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:08.352758 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:08.352738 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:08.519798 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.519623 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zmtdq" event={"ID":"26f26e13-d1c9-4747-914f-af2fb15d7ed7","Type":"ContainerStarted","Data":"ce7a6b813207303e429b93657fd79d8d368fe0747bd56e9bb5b00e98f615a1dc"} Nov 03 20:03:08.521415 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.521386 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dk9sk" event={"ID":"781d2937-5b44-4786-8c19-7629f4548399","Type":"ContainerStarted","Data":"6b24eb598e2a9f9466cd1326bf44ec0fe78cb5cffa557ee6958f14dd294fa9bb"} Nov 03 20:03:08.523697 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.523676 2578 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="34e8aa2323d53ef1700e4a08d936a8eb3de6615c306aca9e16c57afda4261a4d" exitCode=0 Nov 03 20:03:08.523789 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.523744 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerDied","Data":"34e8aa2323d53ef1700e4a08d936a8eb3de6615c306aca9e16c57afda4261a4d"} Nov 03 20:03:08.528455 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.528433 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" event={"ID":"31637ee3-ae67-4836-992c-3553f96a85e5","Type":"ContainerStarted","Data":"8c6bfc4a7a35b9ba8081381e5aecb331af8b21cad3d0dbfdd784900c95748c1b"} Nov 03 20:03:08.559060 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.559007 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zmtdq" podStartSLOduration=1.878321341 podStartE2EDuration="10.558990859s" totalImagesPullingTime="8.680669518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:08.558583946 +0000 UTC m=+10.799344424" watchObservedRunningTime="2026-11-03 20:03:08.558990859 +0000 UTC m=+10.799751334" Nov 03 20:03:08.603476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.603413 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dk9sk" podStartSLOduration=2.741344031 podStartE2EDuration="10.603394738s" totalImagesPullingTime="7.862050707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:08.601052478 +0000 UTC m=+10.841812955" watchObservedRunningTime="2026-11-03 20:03:08.603394738 +0000 UTC m=+10.844155213" Nov 03 20:03:08.645079 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:08.645030 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" podStartSLOduration=2.240233314 podStartE2EDuration="10.645012794s" totalImagesPullingTime="8.40477948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:08.644265467 +0000 UTC m=+10.885025941" watchObservedRunningTime="2026-11-03 20:03:08.645012794 +0000 UTC m=+10.885773269" Nov 03 20:03:09.348811 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:09.348752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:09.348930 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:09.348834 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:09.532542 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:09.532507 2578 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="ac9c900b4479637118deefab19dc27f889daa1150979bb75219da8ed16b86b92" exitCode=0 Nov 03 20:03:09.532988 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:09.532589 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerDied","Data":"ac9c900b4479637118deefab19dc27f889daa1150979bb75219da8ed16b86b92"} Nov 03 20:03:09.534266 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:09.534228 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-947nk" event={"ID":"2bd39722-fd14-42d7-af96-2b37b2bba425","Type":"ContainerStarted","Data":"1bf75efe39afb14e571bb5b9f7c9b9a35fd768190948081140e93afdb76b0133"} Nov 03 20:03:10.349983 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.349461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:10.349983 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.349490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:10.349983 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:10.349603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:10.349983 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:10.349717 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:10.354321 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.354297 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:10.354433 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.354400 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:03:10.355420 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.355393 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:10.394242 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.394200 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-947nk" podStartSLOduration=4.015539827 podStartE2EDuration="12.394188699s" totalImagesPullingTime="8.378648872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:09.61734735 +0000 UTC m=+11.858107827" watchObservedRunningTime="2026-11-03 20:03:10.394188699 +0000 UTC m=+12.634949164" Nov 03 20:03:10.538622 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.538588 2578 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="23762dceb0f91f695644836c0466a2da19f659336078a3e017e257998354aace" exitCode=0 Nov 03 20:03:10.539310 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.538671 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerDied","Data":"23762dceb0f91f695644836c0466a2da19f659336078a3e017e257998354aace"} Nov 03 20:03:10.540515 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.540488 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qptp7" event={"ID":"8ecf92d9-64ba-4348-9124-5f02d8032c9c","Type":"ContainerStarted","Data":"707e4b7fe71c6b00c742d310c8dd08e574e1d5f3880647f9b5df5b60a872df01"} Nov 03 20:03:10.624335 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:10.624250 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qptp7" podStartSLOduration=2.351485843 podStartE2EDuration="12.624237684s" totalImagesPullingTime="10.272751841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:10.624122841 +0000 UTC m=+12.864883315" watchObservedRunningTime="2026-11-03 20:03:10.624237684 +0000 UTC m=+12.864998173" Nov 03 20:03:11.349271 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:11.349243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:11.349434 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:11.349371 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:12.348853 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:12.348825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:12.349334 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:12.348958 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:12.349436 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:12.349415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:12.349552 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:12.349532 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:12.547871 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:12.547836 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerStarted","Data":"a76944204826167b2fab6e5bf6920c0be3acb63d9db1bcda19ad6bc408aafc5b"} Nov 03 20:03:13.349006 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:13.348805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:13.349483 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:13.349123 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:13.553528 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:13.553495 2578 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="a76944204826167b2fab6e5bf6920c0be3acb63d9db1bcda19ad6bc408aafc5b" exitCode=0 Nov 03 20:03:13.553528 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:13.553524 2578 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="99605c645ed759a56daf436dabdcc40f5748c5172e295b7e29922a79c8eaa781" exitCode=0 Nov 03 20:03:13.553727 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:13.553567 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerDied","Data":"a76944204826167b2fab6e5bf6920c0be3acb63d9db1bcda19ad6bc408aafc5b"} Nov 03 20:03:13.553727 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:13.553609 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerDied","Data":"99605c645ed759a56daf436dabdcc40f5748c5172e295b7e29922a79c8eaa781"} Nov 03 20:03:14.348854 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.348833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:14.348953 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:14.348934 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:14.349012 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.348983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:14.349127 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:14.349095 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:14.559765 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.559742 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d45vs" event={"ID":"edd592ba-dbd2-4026-906b-27478b901db5","Type":"ContainerStarted","Data":"6d15de6c42a28db1117ce3aecfc451b7de307c76abd40bf9c91520491ae5ab29"} Nov 03 20:03:14.561834 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.561815 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:14.562154 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.562135 2578 generic.go:356] "Generic (PLEG): container finished" podID="331068c4-2a21-4ab6-9fbf-ea1ce605d767" containerID="56729234d733d9bcc35d964ef7d6d73fb3e93d6b9858ea7820a96bf580eedb8a" exitCode=1 Nov 03 20:03:14.562242 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.562163 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"626decb481da579ff6e8cd469755495cc85ee3ff99f42b685aa4afd9704fdae7"} Nov 03 20:03:14.562242 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.562224 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"c7b5a0537a6dd4ce313fdc60866674078c8f3e697504963bcefe669a4236b765"} Nov 03 20:03:14.562318 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.562267 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerDied","Data":"56729234d733d9bcc35d964ef7d6d73fb3e93d6b9858ea7820a96bf580eedb8a"} Nov 03 20:03:14.562318 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.562298 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"e6ec9927832280da8e37c3848862de33f181980d6ff6e990af6a081a11e63faa"} Nov 03 20:03:14.622810 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.622676 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d45vs" podStartSLOduration=6.706350136 podStartE2EDuration="16.622662966s" totalImagesPullingTime="9.91631283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:14.622637325 +0000 UTC m=+16.863397798" watchObservedRunningTime="2026-11-03 20:03:14.622662966 +0000 UTC m=+16.863423431" Nov 03 20:03:14.948183 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.948159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:14.948360 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:14.948200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:14.948360 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:14.948291 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:14.948360 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:14.948291 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:14.948360 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:14.948333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret podName:25244768-eb42-47ea-a28d-20609fc12f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:30.948321965 +0000 UTC m=+33.189082430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret") pod "global-pull-secret-syncer-kxtqd" (UID: "25244768-eb42-47ea-a28d-20609fc12f42") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:03:14.948360 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:14.948347 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:30.948340794 +0000 UTC m=+33.189101246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:03:15.049367 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:15.049347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:15.049459 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:15.049447 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:03:15.049459 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:15.049459 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:03:15.049527 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:15.049466 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vzpbh for pod openshift-network-diagnostics/network-check-target-7zg6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:15.049527 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:15.049501 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh podName:57170d5a-5e31-40d9-aebd-645e3c12c610 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.049490775 +0000 UTC m=+33.290251241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vzpbh" (UniqueName: "kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh") pod "network-check-target-7zg6w" (UID: "57170d5a-5e31-40d9-aebd-645e3c12c610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:03:15.349299 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:15.349280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:15.349631 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:15.349359 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:15.566913 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:15.566893 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:15.567361 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:15.567339 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"1f146216263cf4317cc69da1854c87088b119f61042077e31c0c481a5d456a5e"} Nov 03 20:03:15.567433 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:15.567419 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"3d52041dc852d1aef21a18b4c129640a6cbad85ecfef7f96ae8d23d628eff203"} Nov 03 20:03:16.348948 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:16.348922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:16.349107 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:16.349046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:16.349107 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:16.349094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:16.349193 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:16.349176 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:17.349209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:17.349181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:17.349751 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:17.349289 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:17.574126 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:17.574102 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:17.574492 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:17.574472 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"45b8c457ac688d3cd81a6ca54ab0368756458adf8444080b95a2c6823bc02842"} Nov 03 20:03:18.349696 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:18.349668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:18.350121 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:18.349749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:18.350121 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:18.349789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:18.350121 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:18.349834 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:19.349570 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.349396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:19.349720 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:19.349644 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:19.581843 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.581820 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:19.582213 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.582193 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"1e304e63eeae38bc22f9b07f8a21886a1fe6d8e1a37f4b82b6a950266852bd02"} Nov 03 20:03:19.582448 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.582432 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:19.582500 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.582489 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:19.582578 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.582564 2578 scope.go:122] "RemoveContainer" containerID="56729234d733d9bcc35d964ef7d6d73fb3e93d6b9858ea7820a96bf580eedb8a" Nov 03 20:03:19.598470 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:19.598451 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:20.348985 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.348956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:20.349196 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:20.349137 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:20.349196 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.349181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:20.349322 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:20.349252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:20.588387 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.588365 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:20.588763 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.588713 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" event={"ID":"331068c4-2a21-4ab6-9fbf-ea1ce605d767","Type":"ContainerStarted","Data":"bc8d72fa2162f03936a496dc1ab78d02da046beca18a3410c2cb77786ed29849"} Nov 03 20:03:20.588916 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.588898 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:20.604470 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.604414 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:20.686982 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.686936 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" podStartSLOduration=7.98941296 podStartE2EDuration="22.686922411s" totalImagesPullingTime="14.697509451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:20.672696351 +0000 UTC m=+22.913456859" watchObservedRunningTime="2026-11-03 20:03:20.686922411 +0000 UTC m=+22.927682884" Nov 03 20:03:20.982519 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.982449 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7zg6w"] Nov 03 20:03:20.982666 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.982623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:20.982802 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:20.982736 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:20.985670 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.985650 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kxtqd"] Nov 03 20:03:20.985761 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.985728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:20.985821 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:20.985807 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:20.986146 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.986130 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8vwxs"] Nov 03 20:03:20.986215 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:20.986205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:20.986303 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:20.986288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:22.349479 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:22.349450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:22.349937 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:22.349542 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:22.349937 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:22.349621 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:22.349937 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:22.349710 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:23.348947 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:23.348920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:23.349099 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:23.349053 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:24.349195 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:24.349166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:24.349195 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:24.349199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:24.349659 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:24.349274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:24.349659 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:24.349343 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:25.349404 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:25.349376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:25.349752 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:25.349477 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vwxs" podUID="2b318e41-0c85-4303-addc-336f59f4a4e7" Nov 03 20:03:26.348779 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:26.348751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:26.348947 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:26.348847 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxtqd" podUID="25244768-eb42-47ea-a28d-20609fc12f42" Nov 03 20:03:26.348947 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:26.348930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:26.349063 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:26.349041 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7zg6w" podUID="57170d5a-5e31-40d9-aebd-645e3c12c610" Nov 03 20:03:27.081002 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.080959 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeReady" Nov 03 20:03:27.081437 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.081095 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:03:27.125062 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.125035 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tc29n"] Nov 03 20:03:27.172266 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.172232 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-htm8c"] Nov 03 20:03:27.173265 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.173241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:27.185534 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.185514 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wh79h"] Nov 03 20:03:27.185670 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.185646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:27.192323 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.192303 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.193996 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.193705 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.205737 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.205718 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7"] Nov 03 20:03:27.205875 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.205854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:27.217269 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.217249 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6l254"] Nov 03 20:03:27.217379 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.217365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.233536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.233520 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:03:27.241616 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.241596 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh"] Nov 03 20:03:27.241705 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.241693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.244648 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.244624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6brv\" (UniqueName: \"kubernetes.io/projected/47ee139e-2eb1-4e93-9536-7c36c003ff40-kube-api-access-g6brv\") pod \"kas-connection-checker-685895c7c9-htm8c\" (UID: \"47ee139e-2eb1-4e93-9536-7c36c003ff40\") " pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:27.244726 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.244664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6z7\" (UniqueName: \"kubernetes.io/projected/aa34540b-36c1-4ef4-b4c9-e14a2419ee31-kube-api-access-7f6z7\") pod \"kas-connection-checker-685895c7c9-tc29n\" (UID: \"aa34540b-36c1-4ef4-b4c9-e14a2419ee31\") " pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:27.263827 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.263809 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4md8j"] Nov 03 20:03:27.263922 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.263909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.282849 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.282833 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9mgnl"] Nov 03 20:03:27.282984 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.282968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.292988 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.292965 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:03:27.293101 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293050 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.293201 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293131 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.293201 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293132 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.293201 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293153 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.293408 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293249 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.293408 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293259 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:03:27.293408 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293340 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:03:27.293408 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.293368 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.297152 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.297134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.297233 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.297160 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d68796f8f-nhd5b"] Nov 03 20:03:27.308604 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.308587 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:03:27.310848 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.310833 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms"] Nov 03 20:03:27.310969 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.310953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.333032 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.332985 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c4d87c5c9-lhjzn"] Nov 03 20:03:27.333145 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.333118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:27.333896 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.333873 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:03:27.333981 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.333893 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:03:27.334039 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.334007 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:03:27.334166 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.334052 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:03:27.334166 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.334054 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:03:27.334166 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.334151 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:03:27.334331 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.334194 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8zgkz\"" Nov 03 20:03:27.334331 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.334206 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:03:27.345859 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.345817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.345946 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.345859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.345946 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.345895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a28d511-b41b-4b0c-8819-6cba4a6620d8-serving-cert\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.345946 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.345921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.346100 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.345956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq5x\" (UniqueName: \"kubernetes.io/projected/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-kube-api-access-fkq5x\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.346100 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0138ff1f-c12a-446e-bd06-e62f8202a476-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.346204 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db5689f3-b955-4d7c-b4b8-c203f7679fa1-config\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.346204 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqkl\" (UniqueName: \"kubernetes.io/projected/0138ff1f-c12a-446e-bd06-e62f8202a476-kube-api-access-4sqkl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.346342 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-tmp\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.346402 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-snapshots\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.346453 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-service-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.346521 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db5689f3-b955-4d7c-b4b8-c203f7679fa1-serving-cert\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.346559 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6brv\" (UniqueName: \"kubernetes.io/projected/47ee139e-2eb1-4e93-9536-7c36c003ff40-kube-api-access-g6brv\") pod \"kas-connection-checker-685895c7c9-htm8c\" (UID: \"47ee139e-2eb1-4e93-9536-7c36c003ff40\") " pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:27.346599 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-serving-cert\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.346599 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.346669 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4p7d\" (UniqueName: \"kubernetes.io/projected/9a28d511-b41b-4b0c-8819-6cba4a6620d8-kube-api-access-h4p7d\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.346767 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zw4h\" (UniqueName: \"kubernetes.io/projected/db5689f3-b955-4d7c-b4b8-c203f7679fa1-kube-api-access-9zw4h\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.346822 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6z7\" (UniqueName: \"kubernetes.io/projected/aa34540b-36c1-4ef4-b4c9-e14a2419ee31-kube-api-access-7f6z7\") pod \"kas-connection-checker-685895c7c9-tc29n\" (UID: \"aa34540b-36c1-4ef4-b4c9-e14a2419ee31\") " pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:27.346911 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-config\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.347001 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.346968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vrx\" (UniqueName: \"kubernetes.io/projected/dfd74d5b-c18f-42cb-be6d-416d2eb1c921-kube-api-access-k2vrx\") pod \"kas-connection-checker-685895c7c9-wh79h\" (UID: \"dfd74d5b-c18f-42cb-be6d-416d2eb1c921\") " pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:27.347056 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.347039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0138ff1f-c12a-446e-bd06-e62f8202a476-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.357122 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.357101 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt"] Nov 03 20:03:27.357204 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.357172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.357255 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.357222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:27.371462 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.371436 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6z7\" (UniqueName: \"kubernetes.io/projected/aa34540b-36c1-4ef4-b4c9-e14a2419ee31-kube-api-access-7f6z7\") pod \"kas-connection-checker-685895c7c9-tc29n\" (UID: \"aa34540b-36c1-4ef4-b4c9-e14a2419ee31\") " pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:27.371539 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.371436 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6brv\" (UniqueName: \"kubernetes.io/projected/47ee139e-2eb1-4e93-9536-7c36c003ff40-kube-api-access-g6brv\") pod \"kas-connection-checker-685895c7c9-htm8c\" (UID: \"47ee139e-2eb1-4e93-9536-7c36c003ff40\") " pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:27.378889 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.378874 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6844b76f4-thtgj"] Nov 03 20:03:27.378981 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.378970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.395138 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395120 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lv272"] Nov 03 20:03:27.395235 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.395303 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395293 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.395391 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395373 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.395445 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.395491 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395447 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.395491 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395460 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.395592 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395551 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:03:27.395592 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395577 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.395688 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.395606 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:03:27.401152 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.401003 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:03:27.401152 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.401112 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:03:27.403477 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.403456 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.403654 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.403635 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.406319 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.406299 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:03:27.409899 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.409883 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:03:27.419407 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.419386 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:27.422916 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.422899 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-59msh"] Nov 03 20:03:27.423028 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.423012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.434169 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434150 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:03:27.434169 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434162 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:03:27.434169 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434151 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:03:27.434308 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434172 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:03:27.434308 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434195 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lf955\"" Nov 03 20:03:27.434308 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434268 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:03:27.434308 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.434294 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\"" Nov 03 20:03:27.435148 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.435134 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tc29n"] Nov 03 20:03:27.435192 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.435159 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-htm8c"] Nov 03 20:03:27.435192 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.435173 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5rssm"] Nov 03 20:03:27.435246 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.435227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:27.439159 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.439142 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:03:27.445827 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.445813 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vl6h5"] Nov 03 20:03:27.445929 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.445915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.447334 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-service-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.447419 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.447419 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.447419 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.447419 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.447419 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.447580 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4p7d\" (UniqueName: \"kubernetes.io/projected/9a28d511-b41b-4b0c-8819-6cba4a6620d8-kube-api-access-h4p7d\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.447580 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zw4h\" (UniqueName: \"kubernetes.io/projected/db5689f3-b955-4d7c-b4b8-c203f7679fa1-kube-api-access-9zw4h\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.447653 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-config\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.447779 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.447833 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.447866 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.447898 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.447930 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a28d511-b41b-4b0c-8819-6cba4a6620d8-serving-cert\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.447930 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.448019 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.447942 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:03:27.448019 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.448007 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert podName:85530c5c-c1ec-4d44-8a76-1ea11f34c7a2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:27.947990524 +0000 UTC m=+30.188750992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9mgnl" (UID: "85530c5c-c1ec-4d44-8a76-1ea11f34c7a2") : secret "networking-console-plugin-cert" not found Nov 03 20:03:27.448144 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-service-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.448225 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.447946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db5689f3-b955-4d7c-b4b8-c203f7679fa1-config\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.448295 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqkl\" (UniqueName: \"kubernetes.io/projected/0138ff1f-c12a-446e-bd06-e62f8202a476-kube-api-access-4sqkl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.448295 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-config\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.448403 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.448403 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq5x\" (UniqueName: \"kubernetes.io/projected/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-kube-api-access-fkq5x\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.448403 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.448540 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.448540 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-tmp\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.448540 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-serving-cert\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.448540 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.448540 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.448540 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.448821 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-snapshots\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.448821 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448565 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db5689f3-b955-4d7c-b4b8-c203f7679fa1-config\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.448821 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448550 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.448821 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.448789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db5689f3-b955-4d7c-b4b8-c203f7679fa1-serving-cert\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.449150 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.449125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-tmp\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.449318 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.449300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.449728 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.449703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-snapshots\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.449820 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.449787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-default-certificate\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.449935 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.449914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.449997 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.449959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7jx\" (UniqueName: \"kubernetes.io/projected/1c49d972-a6cc-488a-90b7-8393e60a93a8-kube-api-access-8m7jx\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.450051 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-stats-auth\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.450128 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nq2q\" (UniqueName: \"kubernetes.io/projected/c2062bc3-f9db-48f9-aedb-b1367774ee26-kube-api-access-4nq2q\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.450128 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vrx\" (UniqueName: \"kubernetes.io/projected/dfd74d5b-c18f-42cb-be6d-416d2eb1c921-kube-api-access-k2vrx\") pod \"kas-connection-checker-685895c7c9-wh79h\" (UID: \"dfd74d5b-c18f-42cb-be6d-416d2eb1c921\") " pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:27.450228 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0138ff1f-c12a-446e-bd06-e62f8202a476-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.450228 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.450228 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.450378 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.450378 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.450378 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0138ff1f-c12a-446e-bd06-e62f8202a476-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450761 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a28d511-b41b-4b0c-8819-6cba4a6620d8-serving-cert\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0138ff1f-c12a-446e-bd06-e62f8202a476-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.450993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.451015 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db5689f3-b955-4d7c-b4b8-c203f7679fa1-serving-cert\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.451019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t28n\" (UniqueName: \"kubernetes.io/projected/a22c4edd-da5c-44d4-a71c-6e162cf7f221-kube-api-access-8t28n\") pod \"volume-data-source-validator-58c9f5f46d-kvjms\" (UID: \"a22c4edd-da5c-44d4-a71c-6e162cf7f221\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.451189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.451409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.451476 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.451444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-serving-cert\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.452345 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.452328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0138ff1f-c12a-446e-bd06-e62f8202a476-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.453689 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453672 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wh79h"] Nov 03 20:03:27.453763 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453717 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7"] Nov 03 20:03:27.453763 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453736 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh"] Nov 03 20:03:27.453763 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453751 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6l254"] Nov 03 20:03:27.453763 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453763 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4md8j"] Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453776 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9mgnl"] Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453792 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms"] Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453811 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-59msh"] Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453831 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d68796f8f-nhd5b"] Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453851 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4d87c5c9-lhjzn"] Nov 03 20:03:27.453885 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453873 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rssm"] Nov 03 20:03:27.454142 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453896 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt"] Nov 03 20:03:27.454142 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453919 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6844b76f4-thtgj"] Nov 03 20:03:27.454142 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453940 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lv272"] Nov 03 20:03:27.454142 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.453958 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vl6h5"] Nov 03 20:03:27.462355 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.462337 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v94zj\"" Nov 03 20:03:27.462511 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.462496 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:03:27.473497 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.473471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zw4h\" (UniqueName: \"kubernetes.io/projected/db5689f3-b955-4d7c-b4b8-c203f7679fa1-kube-api-access-9zw4h\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.473898 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.473882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4p7d\" (UniqueName: \"kubernetes.io/projected/9a28d511-b41b-4b0c-8819-6cba4a6620d8-kube-api-access-h4p7d\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.476797 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.476777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqkl\" (UniqueName: \"kubernetes.io/projected/0138ff1f-c12a-446e-bd06-e62f8202a476-kube-api-access-4sqkl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.477147 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.477130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vrx\" (UniqueName: \"kubernetes.io/projected/dfd74d5b-c18f-42cb-be6d-416d2eb1c921-kube-api-access-k2vrx\") pod \"kas-connection-checker-685895c7c9-wh79h\" (UID: \"dfd74d5b-c18f-42cb-be6d-416d2eb1c921\") " pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:27.477255 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.477233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq5x\" (UniqueName: \"kubernetes.io/projected/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-kube-api-access-fkq5x\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.494086 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494051 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.494379 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494359 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.494454 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494377 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.494454 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494368 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.494454 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494362 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:03:27.494454 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494399 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:03:27.494454 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494446 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:03:27.494738 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.494726 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:03:27.500738 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.500718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:27.513973 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.513956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:27.532201 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.532182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:27.534753 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.534734 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:03:27.534984 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.534966 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5kpvl\"" Nov 03 20:03:27.552587 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.552672 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.552672 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.552672 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.552820 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tb5j\" (UniqueName: \"kubernetes.io/projected/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e-kube-api-access-5tb5j\") pod \"network-check-source-6f78d59f99-59msh\" (UID: \"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:27.552820 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.552820 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.552820 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.552771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e05b9d70-936b-44f5-b0f2-bc68492d7951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.553015 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.552947 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:27.553015 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.552965 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6844b76f4-thtgj: secret "image-registry-tls" not found Nov 03 20:03:27.553131 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxnd\" (UniqueName: \"kubernetes.io/projected/e05b9d70-936b-44f5-b0f2-bc68492d7951-kube-api-access-zhxnd\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.553238 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.553238 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.553238 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.553238 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.553238 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed430c0d-f5a9-4cf9-be18-182b648d8814-cert\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff3150-0ab6-42d2-81ab-b03e7d944674-tmp-dir\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt24\" (UniqueName: \"kubernetes.io/projected/a7ff3150-0ab6-42d2-81ab-b03e7d944674-kube-api-access-9dt24\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-default-certificate\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7jx\" (UniqueName: \"kubernetes.io/projected/1c49d972-a6cc-488a-90b7-8393e60a93a8-kube-api-access-8m7jx\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-stats-auth\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.553466 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq2q\" (UniqueName: \"kubernetes.io/projected/c2062bc3-f9db-48f9-aedb-b1367774ee26-kube-api-access-4nq2q\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.554019 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ff3150-0ab6-42d2-81ab-b03e7d944674-config-volume\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.554019 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.554019 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.554019 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.554019 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.553771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.554270 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.554118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.554750 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.554731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.556005 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.555439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.556005 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.555448 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls podName:eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.055430416 +0000 UTC m=+30.296190884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls") pod "image-registry-6844b76f4-thtgj" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8") : secret "image-registry-tls" not found Nov 03 20:03:27.556178 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.556246 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.556446 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.556513 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:27.556572 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556525 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.556572 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.556529 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:27.556572 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.556555 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4d87c5c9-lhjzn: secret "image-registry-tls" not found Nov 03 20:03:27.556572 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442gc\" (UniqueName: \"kubernetes.io/projected/ed430c0d-f5a9-4cf9-be18-182b648d8814-kube-api-access-442gc\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.556775 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.556775 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.556693 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls podName:259382b1-dd55-47ac-9744-c5b956a13ce7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.056676 +0000 UTC m=+30.297436461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls") pod "image-registry-7c4d87c5c9-lhjzn" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7") : secret "image-registry-tls" not found Nov 03 20:03:27.556879 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.556855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t28n\" (UniqueName: \"kubernetes.io/projected/a22c4edd-da5c-44d4-a71c-6e162cf7f221-kube-api-access-8t28n\") pod \"volume-data-source-validator-58c9f5f46d-kvjms\" (UID: \"a22c4edd-da5c-44d4-a71c-6e162cf7f221\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:27.557042 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.557024 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:03:27.557136 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.557097 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls podName:1c49d972-a6cc-488a-90b7-8393e60a93a8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.057066365 +0000 UTC m=+30.297826831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zqtvt" (UID: "1c49d972-a6cc-488a-90b7-8393e60a93a8") : secret "samples-operator-tls" not found Nov 03 20:03:27.557196 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.557144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.557318 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.557297 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:03:27.557468 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.557379 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.057354704 +0000 UTC m=+30.298115170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : configmap references non-existent config key: service-ca.crt Nov 03 20:03:27.557468 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.557404 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.057392964 +0000 UTC m=+30.298153423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : secret "router-metrics-certs-default" not found Nov 03 20:03:27.557931 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.557895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-default-certificate\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.558572 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.558552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.560547 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.560521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-stats-auth\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.561981 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.561950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.561981 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.561965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.562146 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.562010 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.563683 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.563660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:27.575193 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.575171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:27.579773 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.579741 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.580448 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.580397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7jx\" (UniqueName: \"kubernetes.io/projected/1c49d972-a6cc-488a-90b7-8393e60a93a8-kube-api-access-8m7jx\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:27.580658 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.580637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.581178 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.580880 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t28n\" (UniqueName: \"kubernetes.io/projected/a22c4edd-da5c-44d4-a71c-6e162cf7f221-kube-api-access-8t28n\") pod \"volume-data-source-validator-58c9f5f46d-kvjms\" (UID: \"a22c4edd-da5c-44d4-a71c-6e162cf7f221\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:27.581178 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.580952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:27.581422 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.581353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq2q\" (UniqueName: \"kubernetes.io/projected/c2062bc3-f9db-48f9-aedb-b1367774ee26-kube-api-access-4nq2q\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:27.582147 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.582121 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:27.597204 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.595731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:27.607527 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.607473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:27.659165 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.658184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed430c0d-f5a9-4cf9-be18-182b648d8814-cert\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.659165 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.658261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff3150-0ab6-42d2-81ab-b03e7d944674-tmp-dir\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.659165 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.658294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt24\" (UniqueName: \"kubernetes.io/projected/a7ff3150-0ab6-42d2-81ab-b03e7d944674-kube-api-access-9dt24\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.659165 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.658329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.660558 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.660491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff3150-0ab6-42d2-81ab-b03e7d944674-tmp-dir\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.662327 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.660724 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:27.662327 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.660814 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.160784479 +0000 UTC m=+30.401544931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:27.662327 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.661756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ff3150-0ab6-42d2-81ab-b03e7d944674-config-volume\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.659547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ff3150-0ab6-42d2-81ab-b03e7d944674-config-volume\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.662627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442gc\" (UniqueName: \"kubernetes.io/projected/ed430c0d-f5a9-4cf9-be18-182b648d8814-kube-api-access-442gc\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.662712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tb5j\" (UniqueName: \"kubernetes.io/projected/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e-kube-api-access-5tb5j\") pod \"network-check-source-6f78d59f99-59msh\" (UID: \"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.662751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.662795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e05b9d70-936b-44f5-b0f2-bc68492d7951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.662825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxnd\" (UniqueName: \"kubernetes.io/projected/e05b9d70-936b-44f5-b0f2-bc68492d7951-kube-api-access-zhxnd\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.663575 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:03:27.664315 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.663633 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls podName:a7ff3150-0ab6-42d2-81ab-b03e7d944674 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.163618564 +0000 UTC m=+30.404379033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls") pod "dns-default-5rssm" (UID: "a7ff3150-0ab6-42d2-81ab-b03e7d944674") : secret "dns-default-metrics-tls" not found Nov 03 20:03:27.666907 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.664593 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e05b9d70-936b-44f5-b0f2-bc68492d7951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.666907 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.664690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:27.671111 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.667935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed430c0d-f5a9-4cf9-be18-182b648d8814-cert\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.683883 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.683860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt24\" (UniqueName: \"kubernetes.io/projected/a7ff3150-0ab6-42d2-81ab-b03e7d944674-kube-api-access-9dt24\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:27.693140 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.693041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tb5j\" (UniqueName: \"kubernetes.io/projected/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e-kube-api-access-5tb5j\") pod \"network-check-source-6f78d59f99-59msh\" (UID: \"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:27.694854 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.694806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxnd\" (UniqueName: \"kubernetes.io/projected/e05b9d70-936b-44f5-b0f2-bc68492d7951-kube-api-access-zhxnd\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:27.697613 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.697585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-442gc\" (UniqueName: \"kubernetes.io/projected/ed430c0d-f5a9-4cf9-be18-182b648d8814-kube-api-access-442gc\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.754621 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.754595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:27.801089 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.801047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:27.835613 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.835592 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-htm8c"] Nov 03 20:03:27.838191 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:27.838159 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ee139e_2eb1_4e93_9536_7c36c003ff40.slice/crio-07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c WatchSource:0}: Error finding container 07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c: Status 404 returned error can't find the container with id 07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c Nov 03 20:03:27.839607 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.839570 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tc29n"] Nov 03 20:03:27.842749 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:27.840558 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa34540b_36c1_4ef4_b4c9_e14a2419ee31.slice/crio-dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356 WatchSource:0}: Error finding container dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356: Status 404 returned error can't find the container with id dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356 Nov 03 20:03:27.860782 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.860759 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wh79h"] Nov 03 20:03:27.863985 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:27.863959 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd74d5b_c18f_42cb_be6d_416d2eb1c921.slice/crio-2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2 WatchSource:0}: Error finding container 2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2: Status 404 returned error can't find the container with id 2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2 Nov 03 20:03:27.918348 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.918320 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6l254"] Nov 03 20:03:27.925009 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.923532 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7"] Nov 03 20:03:27.968043 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.967483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:27.968043 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.967667 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:03:27.968043 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:27.967722 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert podName:85530c5c-c1ec-4d44-8a76-1ea11f34c7a2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:28.967708565 +0000 UTC m=+31.208469038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9mgnl" (UID: "85530c5c-c1ec-4d44-8a76-1ea11f34c7a2") : secret "networking-console-plugin-cert" not found Nov 03 20:03:27.975741 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.975722 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh"] Nov 03 20:03:27.987225 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:27.987150 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5689f3_b955_4d7c_b4b8_c203f7679fa1.slice/crio-6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde WatchSource:0}: Error finding container 6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde: Status 404 returned error can't find the container with id 6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde Nov 03 20:03:27.997356 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:27.997334 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4md8j"] Nov 03 20:03:27.999232 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:27.999200 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca30aa7b_65f9_4ccb_83c5_6c555ecb08b5.slice/crio-80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed WatchSource:0}: Error finding container 80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed: Status 404 returned error can't find the container with id 80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed Nov 03 20:03:28.007171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.005890 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms"] Nov 03 20:03:28.009276 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:28.009255 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22c4edd_da5c_44d4_a71c_6e162cf7f221.slice/crio-63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf WatchSource:0}: Error finding container 63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf: Status 404 returned error can't find the container with id 63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf Nov 03 20:03:28.062303 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.062229 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-59msh"] Nov 03 20:03:28.063971 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:28.063951 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0036d1_fc35_4afe_9c9c_f252b3c34b3e.slice/crio-90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170 WatchSource:0}: Error finding container 90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170: Status 404 returned error can't find the container with id 90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170 Nov 03 20:03:28.068590 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.068570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:28.068681 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.068609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:28.068718 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.068660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:28.068763 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068729 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:28.068763 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068740 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.068725221 +0000 UTC m=+31.309485685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : configmap references non-existent config key: service-ca.crt Nov 03 20:03:28.068864 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.068765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:28.068864 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068745 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4d87c5c9-lhjzn: secret "image-registry-tls" not found Nov 03 20:03:28.068864 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.068807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:28.068864 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068816 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:28.068864 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068834 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6844b76f4-thtgj: secret "image-registry-tls" not found Nov 03 20:03:28.068864 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068848 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:03:28.069110 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068848 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls podName:259382b1-dd55-47ac-9744-c5b956a13ce7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.068833759 +0000 UTC m=+31.309594215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls") pod "image-registry-7c4d87c5c9-lhjzn" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7") : secret "image-registry-tls" not found Nov 03 20:03:28.069110 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068876 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:03:28.069110 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068897 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls podName:eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.068882184 +0000 UTC m=+31.309642648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls") pod "image-registry-6844b76f4-thtgj" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8") : secret "image-registry-tls" not found Nov 03 20:03:28.069110 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068917 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls podName:1c49d972-a6cc-488a-90b7-8393e60a93a8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.068907169 +0000 UTC m=+31.309667628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zqtvt" (UID: "1c49d972-a6cc-488a-90b7-8393e60a93a8") : secret "samples-operator-tls" not found Nov 03 20:03:28.069110 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.068944 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.068937347 +0000 UTC m=+31.309697799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : secret "router-metrics-certs-default" not found Nov 03 20:03:28.145161 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.145140 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vl6h5"] Nov 03 20:03:28.147062 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:28.147041 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded430c0d_f5a9_4cf9_be18_182b648d8814.slice/crio-6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097 WatchSource:0}: Error finding container 6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097: Status 404 returned error can't find the container with id 6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097 Nov 03 20:03:28.169917 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.169895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:28.170047 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.170031 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:03:28.170126 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.170035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:28.170126 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.170096 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls podName:a7ff3150-0ab6-42d2-81ab-b03e7d944674 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.170063383 +0000 UTC m=+31.410823852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls") pod "dns-default-5rssm" (UID: "a7ff3150-0ab6-42d2-81ab-b03e7d944674") : secret "dns-default-metrics-tls" not found Nov 03 20:03:28.170237 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.170133 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:28.170237 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.170189 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:29.170173849 +0000 UTC m=+31.410934316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:28.351085 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.351044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:28.351195 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.351159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:28.367418 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.367395 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\"" Nov 03 20:03:28.367521 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.367487 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\"" Nov 03 20:03:28.367554 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.367508 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:03:28.612399 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.611978 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vl6h5" event={"ID":"ed430c0d-f5a9-4cf9-be18-182b648d8814","Type":"ContainerStarted","Data":"6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097"} Nov 03 20:03:28.613785 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.613718 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" event={"ID":"9a28d511-b41b-4b0c-8819-6cba4a6620d8","Type":"ContainerStarted","Data":"9ab7430f768e591d2e24ba01ded4ea75e3859ea2c27dc93fe4ad7c89cdebabbf"} Nov 03 20:03:28.621060 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.620386 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wh79h" event={"ID":"dfd74d5b-c18f-42cb-be6d-416d2eb1c921","Type":"ContainerStarted","Data":"545a67b5a0a49d5a50e1e211fed5b3b1c76e782e3b1dcb7ad50f2adf915f5099"} Nov 03 20:03:28.621060 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.620431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wh79h" event={"ID":"dfd74d5b-c18f-42cb-be6d-416d2eb1c921","Type":"ContainerStarted","Data":"2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2"} Nov 03 20:03:28.624568 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.624498 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" event={"ID":"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e","Type":"ContainerStarted","Data":"90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170"} Nov 03 20:03:28.626776 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.626751 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4md8j" event={"ID":"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5","Type":"ContainerStarted","Data":"80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed"} Nov 03 20:03:28.628570 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.628541 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" event={"ID":"db5689f3-b955-4d7c-b4b8-c203f7679fa1","Type":"ContainerStarted","Data":"6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde"} Nov 03 20:03:28.630135 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.630104 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-htm8c" event={"ID":"47ee139e-2eb1-4e93-9536-7c36c003ff40","Type":"ContainerStarted","Data":"ea12be601e228ee0e41059b0db91e514b9579a2dd2ec6ddc35f0bfb4c24b367e"} Nov 03 20:03:28.630221 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.630157 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-htm8c" event={"ID":"47ee139e-2eb1-4e93-9536-7c36c003ff40","Type":"ContainerStarted","Data":"07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c"} Nov 03 20:03:28.631410 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.631388 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" event={"ID":"0138ff1f-c12a-446e-bd06-e62f8202a476","Type":"ContainerStarted","Data":"55c3754192e88cd9b36ef2b7713e68b41da83840a1cb25380601a5b3c5d5b8e9"} Nov 03 20:03:28.635125 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.635104 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tc29n" event={"ID":"aa34540b-36c1-4ef4-b4c9-e14a2419ee31","Type":"ContainerStarted","Data":"25d7f39a0ee3840988c3a66750611f42c3ce06fb55f1b4b63e96357981d8807a"} Nov 03 20:03:28.635209 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.635142 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tc29n" event={"ID":"aa34540b-36c1-4ef4-b4c9-e14a2419ee31","Type":"ContainerStarted","Data":"dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356"} Nov 03 20:03:28.637322 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.637285 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" event={"ID":"a22c4edd-da5c-44d4-a71c-6e162cf7f221","Type":"ContainerStarted","Data":"63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf"} Nov 03 20:03:28.661210 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.661165 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wh79h" podStartSLOduration=418.661147834 podStartE2EDuration="6m58.661147834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:28.661052766 +0000 UTC m=+30.901813258" watchObservedRunningTime="2026-11-03 20:03:28.661147834 +0000 UTC m=+30.901908303" Nov 03 20:03:28.701185 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.701135 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-htm8c" podStartSLOduration=418.701118211 podStartE2EDuration="6m58.701118211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:28.700754246 +0000 UTC m=+30.941514721" watchObservedRunningTime="2026-11-03 20:03:28.701118211 +0000 UTC m=+30.941878686" Nov 03 20:03:28.746028 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.745983 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tc29n" podStartSLOduration=418.745967023 podStartE2EDuration="6m58.745967023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:28.744400589 +0000 UTC m=+30.985161064" watchObservedRunningTime="2026-11-03 20:03:28.745967023 +0000 UTC m=+30.986727498" Nov 03 20:03:28.979189 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:28.979111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:28.979329 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.979275 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:03:28.979373 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:28.979353 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert podName:85530c5c-c1ec-4d44-8a76-1ea11f34c7a2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:30.979333456 +0000 UTC m=+33.220093925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9mgnl" (UID: "85530c5c-c1ec-4d44-8a76-1ea11f34c7a2") : secret "networking-console-plugin-cert" not found Nov 03 20:03:29.079913 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.079860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:29.080110 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.079934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:29.080110 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.079993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:29.080110 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.080036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:29.080289 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.080175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:29.080345 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080329 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:29.080345 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080343 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4d87c5c9-lhjzn: secret "image-registry-tls" not found Nov 03 20:03:29.080447 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls podName:259382b1-dd55-47ac-9744-c5b956a13ce7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.08038208 +0000 UTC m=+33.321142536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls") pod "image-registry-7c4d87c5c9-lhjzn" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7") : secret "image-registry-tls" not found Nov 03 20:03:29.080785 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080761 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.080750537 +0000 UTC m=+33.321511003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : configmap references non-existent config key: service-ca.crt Nov 03 20:03:29.080903 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080817 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:29.080903 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080825 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6844b76f4-thtgj: secret "image-registry-tls" not found Nov 03 20:03:29.080903 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080846 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls podName:eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.080839649 +0000 UTC m=+33.321600114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls") pod "image-registry-6844b76f4-thtgj" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8") : secret "image-registry-tls" not found Nov 03 20:03:29.080903 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080902 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:03:29.081048 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080926 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls podName:1c49d972-a6cc-488a-90b7-8393e60a93a8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.080920351 +0000 UTC m=+33.321680816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zqtvt" (UID: "1c49d972-a6cc-488a-90b7-8393e60a93a8") : secret "samples-operator-tls" not found Nov 03 20:03:29.081048 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080969 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:03:29.081048 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.080997 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.080987726 +0000 UTC m=+33.321748202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : secret "router-metrics-certs-default" not found Nov 03 20:03:29.180792 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.180757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:29.181452 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:29.180906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:29.181452 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.181096 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:03:29.181452 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.181145 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls podName:a7ff3150-0ab6-42d2-81ab-b03e7d944674 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.181133669 +0000 UTC m=+33.421894133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls") pod "dns-default-5rssm" (UID: "a7ff3150-0ab6-42d2-81ab-b03e7d944674") : secret "dns-default-metrics-tls" not found Nov 03 20:03:29.181452 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.181196 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:29.181452 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:29.181259 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:31.181242879 +0000 UTC m=+33.422003344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:30.650118 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:30.650060 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" event={"ID":"a22c4edd-da5c-44d4-a71c-6e162cf7f221","Type":"ContainerStarted","Data":"baeb181add27c7890e8f1c9077d3d665eefcc87281dec498e06f30c970fe7ac3"} Nov 03 20:03:30.690793 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:30.690743 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" podStartSLOduration=391.811118356 podStartE2EDuration="6m33.690722301s" totalImagesPullingTime="1.879603945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:30.689834179 +0000 UTC m=+32.930594654" watchObservedRunningTime="2026-11-03 20:03:30.690722301 +0000 UTC m=+32.931482776" Nov 03 20:03:30.777778 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:30.777742 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:30.863780 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:30.863740 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.004571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.004674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.004707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.004888 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.004956 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert podName:85530c5c-c1ec-4d44-8a76-1ea11f34c7a2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.004937123 +0000 UTC m=+37.245697588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9mgnl" (UID: "85530c5c-c1ec-4d44-8a76-1ea11f34c7a2") : secret "networking-console-plugin-cert" not found Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.005714 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:03:31.005794 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.005765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:03.005749889 +0000 UTC m=+65.246510356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : secret "metrics-daemon-secret" not found Nov 03 20:03:31.008717 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.008130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:31.085505 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.085305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:31.105571 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.105544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:31.105728 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.105592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:31.105728 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.105640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:31.105728 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.105702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:31.105896 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.105747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:31.105896 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.105799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106128 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106152 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6844b76f4-thtgj: secret "image-registry-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106214 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls podName:eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.106194496 +0000 UTC m=+37.346954969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls") pod "image-registry-6844b76f4-thtgj" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8") : secret "image-registry-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106595 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106610 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4d87c5c9-lhjzn: secret "image-registry-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106656 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls podName:259382b1-dd55-47ac-9744-c5b956a13ce7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.106641446 +0000 UTC m=+37.347401911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls") pod "image-registry-7c4d87c5c9-lhjzn" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7") : secret "image-registry-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106708 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106726 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.106713576 +0000 UTC m=+37.347474041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : configmap references non-existent config key: service-ca.crt Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106774 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls podName:1c49d972-a6cc-488a-90b7-8393e60a93a8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.106756527 +0000 UTC m=+37.347517003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zqtvt" (UID: "1c49d972-a6cc-488a-90b7-8393e60a93a8") : secret "samples-operator-tls" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106794 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:03:31.106850 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.106823 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.106814376 +0000 UTC m=+37.347574854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : secret "router-metrics-certs-default" not found Nov 03 20:03:31.109013 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.108991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:31.206732 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.206699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:31.206856 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.206832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:31.207619 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.206977 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:31.207619 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.207041 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.20702121 +0000 UTC m=+37.447781677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:31.207619 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.207535 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:03:31.207619 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:31.207595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls podName:a7ff3150-0ab6-42d2-81ab-b03e7d944674 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:35.207579427 +0000 UTC m=+37.448339897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls") pod "dns-default-5rssm" (UID: "a7ff3150-0ab6-42d2-81ab-b03e7d944674") : secret "dns-default-metrics-tls" not found Nov 03 20:03:31.401220 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.401168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:31.442126 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.440735 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kxtqd"] Nov 03 20:03:31.656778 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.656633 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kxtqd" event={"ID":"25244768-eb42-47ea-a28d-20609fc12f42","Type":"ContainerStarted","Data":"eb9b3af502ed6e1dd7485719d8f06c9dd4567f2a7d96b2c4b6e6565197138fbc"} Nov 03 20:03:31.754325 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:31.754235 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7zg6w"] Nov 03 20:03:31.758288 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:31.758256 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57170d5a_5e31_40d9_aebd_645e3c12c610.slice/crio-60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7 WatchSource:0}: Error finding container 60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7: Status 404 returned error can't find the container with id 60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7 Nov 03 20:03:32.049003 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.048915 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:32.661362 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.661287 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7zg6w" event={"ID":"57170d5a-5e31-40d9-aebd-645e3c12c610","Type":"ContainerStarted","Data":"60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7"} Nov 03 20:03:32.663317 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.663270 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4md8j" event={"ID":"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5","Type":"ContainerStarted","Data":"65c7389b831e14b43dd73550000402ea93b50f6cec6d16fbf5ffc53264e40ef8"} Nov 03 20:03:32.665563 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.665542 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/0.log" Nov 03 20:03:32.665651 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.665589 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" containerID="1c402047586153ef641a8eaa224cf2542d3279509faef325497139dbc6c2329f" exitCode=255 Nov 03 20:03:32.665651 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.665622 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" event={"ID":"9a28d511-b41b-4b0c-8819-6cba4a6620d8","Type":"ContainerDied","Data":"1c402047586153ef641a8eaa224cf2542d3279509faef325497139dbc6c2329f"} Nov 03 20:03:32.665849 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.665833 2578 scope.go:122] "RemoveContainer" containerID="1c402047586153ef641a8eaa224cf2542d3279509faef325497139dbc6c2329f" Nov 03 20:03:32.707207 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:32.707155 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4md8j" podStartSLOduration=388.368927644 podStartE2EDuration="6m32.707114304s" totalImagesPullingTime="4.33818666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:32.705660995 +0000 UTC m=+34.946421479" watchObservedRunningTime="2026-11-03 20:03:32.707114304 +0000 UTC m=+34.947874778" Nov 03 20:03:33.119961 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.118545 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2nkf6"] Nov 03 20:03:33.133244 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.132406 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.150145 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.149775 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:03:33.150145 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.149853 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:03:33.150145 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.150035 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-86559\"" Nov 03 20:03:33.151125 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.151104 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nkf6"] Nov 03 20:03:33.218297 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.218239 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-c6zf8-lzlj9"] Nov 03 20:03:33.226119 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.226044 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:33.229357 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.229334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18f94909-ffd1-4bea-ad03-10772d7775f3-crio-socket\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.229485 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.229369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjk6w\" (UniqueName: \"kubernetes.io/projected/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-api-access-pjk6w\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.229546 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.229507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f94909-ffd1-4bea-ad03-10772d7775f3-data-volume\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.229595 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.229546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.229644 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.229623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.250112 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.242229 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c6zf8-lzlj9"] Nov 03 20:03:33.250112 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.242369 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.261172 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.261148 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-95qsx\"" Nov 03 20:03:33.330098 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f94909-ffd1-4bea-ad03-10772d7775f3-data-volume\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330098 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.330267 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330267 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330267 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.330267 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18f94909-ffd1-4bea-ad03-10772d7775f3-crio-socket\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330267 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjk6w\" (UniqueName: \"kubernetes.io/projected/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-api-access-pjk6w\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330417 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.330417 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f94909-ffd1-4bea-ad03-10772d7775f3-data-volume\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330417 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.330417 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.330383 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:03:33.330530 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.330449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:33.830435701 +0000 UTC m=+36.071196172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : configmap "kube-rbac-proxy" not found Nov 03 20:03:33.330530 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.330469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18f94909-ffd1-4bea-ad03-10772d7775f3-crio-socket\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.330530 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.330501 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:33.330530 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.330525 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:33.830517507 +0000 UTC m=+36.071277996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:33.356223 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.356195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjk6w\" (UniqueName: \"kubernetes.io/projected/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-api-access-pjk6w\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.431154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.431240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.431325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.431355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.432147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.432775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.433034 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.433003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.468329 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.468242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.581717 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.579367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:33.673286 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.672730 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vl6h5" event={"ID":"ed430c0d-f5a9-4cf9-be18-182b648d8814","Type":"ContainerStarted","Data":"4684bdb3792e8145cdec593d76a4e856b015d3aa5aa143c24a9973b6c46de82f"} Nov 03 20:03:33.675518 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.675494 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/1.log" Nov 03 20:03:33.678254 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.678234 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/0.log" Nov 03 20:03:33.678366 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.678273 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" exitCode=255 Nov 03 20:03:33.678718 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.678534 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" event={"ID":"9a28d511-b41b-4b0c-8819-6cba4a6620d8","Type":"ContainerDied","Data":"da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7"} Nov 03 20:03:33.678718 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.678586 2578 scope.go:122] "RemoveContainer" containerID="1c402047586153ef641a8eaa224cf2542d3279509faef325497139dbc6c2329f" Nov 03 20:03:33.678718 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.678640 2578 scope.go:122] "RemoveContainer" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" Nov 03 20:03:33.678934 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.678850 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6l254_openshift-console-operator(9a28d511-b41b-4b0c-8819-6cba4a6620d8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" Nov 03 20:03:33.681616 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.680987 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" event={"ID":"db5689f3-b955-4d7c-b4b8-c203f7679fa1","Type":"ContainerStarted","Data":"e5021b488fdc03f9ab1d2eaf213e4c49ab094d230767cdc687585ff49fea00bd"} Nov 03 20:03:33.685182 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.685134 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" event={"ID":"0138ff1f-c12a-446e-bd06-e62f8202a476","Type":"ContainerStarted","Data":"184fa9e8dfa22b604fdb8860c7bd00a0dbd455df4359730ffae5e4f87b93a70b"} Nov 03 20:03:33.722244 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.722082 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vl6h5" podStartSLOduration=1.543419858 podStartE2EDuration="6.722049661s" totalImagesPullingTime="5.178629803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:33.720823384 +0000 UTC m=+35.961583861" watchObservedRunningTime="2026-11-03 20:03:33.722049661 +0000 UTC m=+35.962810136" Nov 03 20:03:33.789534 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.789448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:33.806140 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.806059 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" podStartSLOduration=388.962819055 podStartE2EDuration="6m33.806040817s" totalImagesPullingTime="4.843221762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:33.762914015 +0000 UTC m=+36.003674489" watchObservedRunningTime="2026-11-03 20:03:33.806040817 +0000 UTC m=+36.046801292" Nov 03 20:03:33.806289 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.806219 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" podStartSLOduration=387.003221903 podStartE2EDuration="6m31.806207219s" totalImagesPullingTime="4.802985316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:33.806017969 +0000 UTC m=+36.046778444" watchObservedRunningTime="2026-11-03 20:03:33.806207219 +0000 UTC m=+36.046967694" Nov 03 20:03:33.835556 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.835513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.835697 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.835609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:33.837841 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.837005 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:03:33.837841 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.837545 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:33.837841 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.837756 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:34.837055205 +0000 UTC m=+37.077815672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : configmap "kube-rbac-proxy" not found Nov 03 20:03:33.837841 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:33.837805 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:34.837778591 +0000 UTC m=+37.078539043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:33.904957 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.904924 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:33.920390 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.920363 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td"] Nov 03 20:03:33.939761 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.939651 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt"] Nov 03 20:03:33.940694 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.940672 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:33.958536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.958337 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:33.958536 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.958501 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t7mgh\"" Nov 03 20:03:33.959412 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.959385 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:03:33.985913 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.985874 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c6zf8-lzlj9"] Nov 03 20:03:33.986022 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.985925 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td"] Nov 03 20:03:33.986022 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.985940 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt"] Nov 03 20:03:33.986022 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:33.986007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:34.012610 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.009374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:34.036978 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.036953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtq4\" (UniqueName: \"kubernetes.io/projected/e341cc19-7823-4078-bd1c-1a7d3732882d-kube-api-access-9xtq4\") pod \"migrator-66df5d496b-qv7td\" (UID: \"e341cc19-7823-4078-bd1c-1a7d3732882d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:34.105638 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.105511 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:34.144461 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.143618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/967ed630-92ed-447b-b106-bb39959cbf0b-kube-api-access-v2l6w\") pod \"migrator-66df5d496b-msbdt\" (UID: \"967ed630-92ed-447b-b106-bb39959cbf0b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:34.144461 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.143707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtq4\" (UniqueName: \"kubernetes.io/projected/e341cc19-7823-4078-bd1c-1a7d3732882d-kube-api-access-9xtq4\") pod \"migrator-66df5d496b-qv7td\" (UID: \"e341cc19-7823-4078-bd1c-1a7d3732882d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:34.164474 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.164450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtq4\" (UniqueName: \"kubernetes.io/projected/e341cc19-7823-4078-bd1c-1a7d3732882d-kube-api-access-9xtq4\") pod \"migrator-66df5d496b-qv7td\" (UID: \"e341cc19-7823-4078-bd1c-1a7d3732882d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:34.245212 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.244818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/967ed630-92ed-447b-b106-bb39959cbf0b-kube-api-access-v2l6w\") pod \"migrator-66df5d496b-msbdt\" (UID: \"967ed630-92ed-447b-b106-bb39959cbf0b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:34.266999 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.266972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/967ed630-92ed-447b-b106-bb39959cbf0b-kube-api-access-v2l6w\") pod \"migrator-66df5d496b-msbdt\" (UID: \"967ed630-92ed-447b-b106-bb39959cbf0b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:34.286651 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.286469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:34.294405 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.294384 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xvg7r"] Nov 03 20:03:34.324454 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.324242 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:34.328031 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.327190 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xvg7r"] Nov 03 20:03:34.328031 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.327297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.344329 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.344228 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cpwrs\"" Nov 03 20:03:34.344329 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.344298 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:03:34.420321 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.420288 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:03:34.420523 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.420507 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:03:34.420589 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.420579 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:03:34.447085 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.447040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70625344-a07f-4396-8843-f66a9f5421d1-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.447205 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.447103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70625344-a07f-4396-8843-f66a9f5421d1-signing-key\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.447284 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.447262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtdg\" (UniqueName: \"kubernetes.io/projected/70625344-a07f-4396-8843-f66a9f5421d1-kube-api-access-ddtdg\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.548344 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.548266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtdg\" (UniqueName: \"kubernetes.io/projected/70625344-a07f-4396-8843-f66a9f5421d1-kube-api-access-ddtdg\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.548482 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.548375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70625344-a07f-4396-8843-f66a9f5421d1-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.548482 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.548400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70625344-a07f-4396-8843-f66a9f5421d1-signing-key\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.549216 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.549165 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70625344-a07f-4396-8843-f66a9f5421d1-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.551335 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.551308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70625344-a07f-4396-8843-f66a9f5421d1-signing-key\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.569457 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.569429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtdg\" (UniqueName: \"kubernetes.io/projected/70625344-a07f-4396-8843-f66a9f5421d1-kube-api-access-ddtdg\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.608725 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.608600 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td"] Nov 03 20:03:34.611509 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:34.611481 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode341cc19_7823_4078_bd1c_1a7d3732882d.slice/crio-51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c WatchSource:0}: Error finding container 51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c: Status 404 returned error can't find the container with id 51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c Nov 03 20:03:34.668766 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.668739 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:34.677272 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.677154 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt"] Nov 03 20:03:34.680360 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:34.680333 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967ed630_92ed_447b_b106_bb39959cbf0b.slice/crio-12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2 WatchSource:0}: Error finding container 12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2: Status 404 returned error can't find the container with id 12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2 Nov 03 20:03:34.689792 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.689766 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" event={"ID":"e341cc19-7823-4078-bd1c-1a7d3732882d","Type":"ContainerStarted","Data":"51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c"} Nov 03 20:03:34.691609 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.691585 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/1.log" Nov 03 20:03:34.691995 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.691978 2578 scope.go:122] "RemoveContainer" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" Nov 03 20:03:34.692207 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:34.692183 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6l254_openshift-console-operator(9a28d511-b41b-4b0c-8819-6cba4a6620d8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" Nov 03 20:03:34.694101 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.693890 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" event={"ID":"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e","Type":"ContainerStarted","Data":"f250a11177921b45d93dd99b72ba628784fe316eb4f8bb52dcc1d51f1ca9f16d"} Nov 03 20:03:34.695352 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.695303 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" event={"ID":"967ed630-92ed-447b-b106-bb39959cbf0b","Type":"ContainerStarted","Data":"12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2"} Nov 03 20:03:34.698304 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.698281 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" event={"ID":"aa7644a3-9a1a-47a8-82da-44de32fa7ee2","Type":"ContainerStarted","Data":"fe3adbd709414b43e3046dfce686930becb6b485cafc2ac8e0a451c049582c99"} Nov 03 20:03:34.698370 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.698321 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" event={"ID":"aa7644a3-9a1a-47a8-82da-44de32fa7ee2","Type":"ContainerStarted","Data":"0e2dbcbefeaf5f027961da4c542791e61adb71a3b3ffa4c239d98f6a35617874"} Nov 03 20:03:34.719266 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.719241 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:34.791439 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.790856 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" podStartSLOduration=1.790835751 podStartE2EDuration="1.790835751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:34.789384413 +0000 UTC m=+37.030144887" watchObservedRunningTime="2026-11-03 20:03:34.790835751 +0000 UTC m=+37.031596229" Nov 03 20:03:34.811252 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.811140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:34.833156 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.833111 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" podStartSLOduration=384.696151285 podStartE2EDuration="6m30.833098997s" totalImagesPullingTime="6.136947712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:34.832283737 +0000 UTC m=+37.073044209" watchObservedRunningTime="2026-11-03 20:03:34.833098997 +0000 UTC m=+37.073859470" Nov 03 20:03:34.853735 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.853251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:34.853735 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:34.853332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:34.853735 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:34.853349 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:34.853735 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:34.853391 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:36.853379122 +0000 UTC m=+39.094139590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:34.853735 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:34.853448 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:03:34.853735 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:34.853487 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:36.853475519 +0000 UTC m=+39.094235987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : configmap "kube-rbac-proxy" not found Nov 03 20:03:35.012343 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.012254 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xvg7r"] Nov 03 20:03:35.013889 ip-10-0-140-102 kubenswrapper[2578]: W1103 20:03:35.013860 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70625344_a07f_4396_8843_f66a9f5421d1.slice/crio-88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16 WatchSource:0}: Error finding container 88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16: Status 404 returned error can't find the container with id 88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16 Nov 03 20:03:35.053043 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.053012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:35.056573 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.056549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:35.056702 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.056678 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:03:35.056805 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.056737 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert podName:85530c5c-c1ec-4d44-8a76-1ea11f34c7a2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.056721448 +0000 UTC m=+45.297481917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9mgnl" (UID: "85530c5c-c1ec-4d44-8a76-1ea11f34c7a2") : secret "networking-console-plugin-cert" not found Nov 03 20:03:35.157554 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.157522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:35.157737 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.157595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:35.157737 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.157637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:35.157737 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.157672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:35.157944 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.157922 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:03:35.158031 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.157923 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:35.158031 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.157972 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6844b76f4-thtgj: secret "image-registry-tls" not found Nov 03 20:03:35.158031 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.157961 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.157940686 +0000 UTC m=+45.398701140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : configmap references non-existent config key: service-ca.crt Nov 03 20:03:35.158031 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158011 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:03:35.158205 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.158043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:35.158205 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158181 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:03:35.158205 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158193 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4d87c5c9-lhjzn: secret "image-registry-tls" not found Nov 03 20:03:35.158301 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158232 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls podName:259382b1-dd55-47ac-9744-c5b956a13ce7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.158214252 +0000 UTC m=+45.398974704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls") pod "image-registry-7c4d87c5c9-lhjzn" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7") : secret "image-registry-tls" not found Nov 03 20:03:35.158301 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158246 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls podName:1c49d972-a6cc-488a-90b7-8393e60a93a8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.158239424 +0000 UTC m=+45.398999940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zqtvt" (UID: "1c49d972-a6cc-488a-90b7-8393e60a93a8") : secret "samples-operator-tls" not found Nov 03 20:03:35.158301 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158256 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls podName:eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.158251619 +0000 UTC m=+45.399012071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls") pod "image-registry-6844b76f4-thtgj" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8") : secret "image-registry-tls" not found Nov 03 20:03:35.158423 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.158329 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs podName:c2062bc3-f9db-48f9-aedb-b1367774ee26 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.15830792 +0000 UTC m=+45.399068387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs") pod "router-default-7d68796f8f-nhd5b" (UID: "c2062bc3-f9db-48f9-aedb-b1367774ee26") : secret "router-metrics-certs-default" not found Nov 03 20:03:35.259255 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.259228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:35.259422 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.259322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:35.259422 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.259383 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:03:35.259544 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.259450 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls podName:a7ff3150-0ab6-42d2-81ab-b03e7d944674 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.259429885 +0000 UTC m=+45.500190350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls") pod "dns-default-5rssm" (UID: "a7ff3150-0ab6-42d2-81ab-b03e7d944674") : secret "dns-default-metrics-tls" not found Nov 03 20:03:35.259544 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.259486 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:35.259654 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:35.259546 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.259530695 +0000 UTC m=+45.500291163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:35.704912 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.704419 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" event={"ID":"70625344-a07f-4396-8843-f66a9f5421d1","Type":"ContainerStarted","Data":"7b6624aed1716c49d45e963dff0789883e7084151bc55db142fc7bdf706ef233"} Nov 03 20:03:35.704912 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.704535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" event={"ID":"70625344-a07f-4396-8843-f66a9f5421d1","Type":"ContainerStarted","Data":"88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16"} Nov 03 20:03:35.706694 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.706220 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7zg6w" event={"ID":"57170d5a-5e31-40d9-aebd-645e3c12c610","Type":"ContainerStarted","Data":"e0995788ab50883340f6a844b7ab8e93bbaa53a7279a3d0180d4096e4d6782c2"} Nov 03 20:03:35.706694 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.706491 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:35.747848 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.747795 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" podStartSLOduration=1.747780807 podStartE2EDuration="1.747780807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:35.7459989 +0000 UTC m=+37.986759374" watchObservedRunningTime="2026-11-03 20:03:35.747780807 +0000 UTC m=+37.988541281" Nov 03 20:03:35.791438 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.790125 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7zg6w" podStartSLOduration=34.234056789 podStartE2EDuration="37.790110771s" totalImagesPullingTime="3.556053982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:35.789459809 +0000 UTC m=+38.030220296" watchObservedRunningTime="2026-11-03 20:03:35.790110771 +0000 UTC m=+38.030871245" Nov 03 20:03:35.906236 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:35.906114 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:36.078503 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.078458 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:36.256239 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.256206 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:36.344754 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.344607 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:36.442947 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.442861 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:36.875865 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.875825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:36.876255 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.875977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:36.876255 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:36.875991 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:36.876255 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:36.876115 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:40.876094551 +0000 UTC m=+43.116855022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:36.876491 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:36.876032 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:03:36.876491 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:36.876322 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:40.876307433 +0000 UTC m=+43.117067900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : configmap "kube-rbac-proxy" not found Nov 03 20:03:36.930171 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:36.930137 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:37.427045 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.427009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:37.430610 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.430585 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zmtdq_26f26e13-d1c9-4747-914f-af2fb15d7ed7/dns-node-resolver/0.log" Nov 03 20:03:37.522359 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.522326 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:37.575964 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.575936 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:37.576121 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.575998 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:37.576435 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.576419 2578 scope.go:122] "RemoveContainer" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" Nov 03 20:03:37.576638 ip-10-0-140-102 kubenswrapper[2578]: E1103 20:03:37.576616 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6l254_openshift-console-operator(9a28d511-b41b-4b0c-8819-6cba4a6620d8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" Nov 03 20:03:37.710555 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.710459 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:37.714095 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.714050 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" event={"ID":"967ed630-92ed-447b-b106-bb39959cbf0b","Type":"ContainerStarted","Data":"90bb38f9586eeed0ddb7ce07216c8deffef279ffc245e3ae4481ec9182c7b1fa"} Nov 03 20:03:37.715512 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.715492 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" event={"ID":"e341cc19-7823-4078-bd1c-1a7d3732882d","Type":"ContainerStarted","Data":"38508cf4e9007812b37a34189268511adb6fcc5061cc9f505109601b84bac266"} Nov 03 20:03:37.757395 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.757356 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" podStartSLOduration=2.659307398 podStartE2EDuration="4.757344987s" totalImagesPullingTime="2.098037589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:37.756489136 +0000 UTC m=+39.997249610" watchObservedRunningTime="2026-11-03 20:03:37.757344987 +0000 UTC m=+39.998105460" Nov 03 20:03:37.797756 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.797723 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" podStartSLOduration=2.626404632 podStartE2EDuration="4.797713159s" totalImagesPullingTime="2.171308527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:37.797624832 +0000 UTC m=+40.038385322" watchObservedRunningTime="2026-11-03 20:03:37.797713159 +0000 UTC m=+40.038473632" Nov 03 20:03:37.827673 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:37.827640 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:38.031241 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:38.031191 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bkb5w_fff6f118-0ff9-4478-b759-04e2d3f63420/node-ca/0.log" Nov 03 20:03:38.033761 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:38.033736 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:38.218806 ip-10-0-140-102 kubenswrapper[2578]: I1103 20:03:38.218778 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:38.424700 ip-10-0-140-102 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:03:38.446415 ip-10-0-140-102 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:03:38.446576 ip-10-0-140-102 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:03:38.446622 ip-10-0-140-102 systemd[1]: kubelet.service: Consumed 3.120s CPU time. Nov 03 20:03:38.467793 ip-10-0-140-102 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:03:38.581016 ip-10-0-140-102 kubenswrapper[5043]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:03:38.581016 ip-10-0-140-102 kubenswrapper[5043]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:03:38.581016 ip-10-0-140-102 kubenswrapper[5043]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:03:38.581390 ip-10-0-140-102 kubenswrapper[5043]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:03:38.584264 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584180 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584266 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584272 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584284 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584288 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584293 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584297 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584303 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584307 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584311 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584315 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584321 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.584320 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584326 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584330 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584334 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584339 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584350 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584355 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584359 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584363 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584367 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584372 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584376 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584380 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584384 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584388 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584392 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584396 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584401 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584405 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584409 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584413 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.584621 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584417 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584421 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584425 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584429 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584434 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584438 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584443 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584447 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584451 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584455 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584459 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584464 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584468 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584473 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584477 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584482 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584486 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584493 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584500 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.585125 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584506 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584511 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584516 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584520 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584525 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584529 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584533 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584537 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584541 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584545 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584549 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584553 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584557 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584561 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584567 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584571 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584575 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584579 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584583 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584587 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.585597 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584591 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584595 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584599 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584603 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584607 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584611 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584616 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584620 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584625 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584629 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584633 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584638 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584641 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584646 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584650 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584654 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584658 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584662 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584665 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.586114 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584669 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584674 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584679 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584683 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584687 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584691 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584696 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584700 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584704 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584708 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584712 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584716 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584724 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584730 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584734 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.584739 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585857 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585866 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585870 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.586579 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585875 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585879 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585884 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585888 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585893 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585897 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585901 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585906 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585910 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585913 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585918 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585922 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585926 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585931 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585935 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585939 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585943 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585947 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585951 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585956 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.587027 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585960 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585964 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585968 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585972 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585976 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585979 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585983 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585987 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585991 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585995 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.585999 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586003 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586007 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586012 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586016 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586021 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586025 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586029 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586033 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586037 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.587567 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586041 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586046 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586050 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586054 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586058 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586063 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586083 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586088 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586092 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586096 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586100 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586104 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586108 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586113 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586116 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586121 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586125 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586129 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586133 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.588039 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586137 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586141 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586145 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586150 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586153 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586158 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586162 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586170 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586176 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586181 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586186 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586190 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586194 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586198 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586203 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586207 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586211 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586216 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586220 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.588557 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586224 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586228 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586232 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586236 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586240 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586245 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586249 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586253 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586257 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586261 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586265 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586269 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586273 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586280 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586285 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586289 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586293 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586299 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586303 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.589029 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586307 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586311 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586315 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586319 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586324 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.586328 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587121 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587130 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587137 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587142 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587148 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587152 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587157 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587162 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587166 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587173 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587177 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587181 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587185 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587189 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.589491 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587193 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587196 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587201 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587205 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587211 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587217 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587222 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587226 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587230 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587234 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587238 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587241 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587245 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587249 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587253 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587257 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587262 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587266 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587270 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587274 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.589973 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587278 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587282 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587287 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587291 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587295 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587299 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587303 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587310 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587314 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587318 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587322 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587325 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587329 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587333 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587337 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587341 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587345 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587350 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587354 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587358 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.590465 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587362 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587366 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587369 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587373 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587377 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587381 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587386 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587389 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587393 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587398 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587402 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587406 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587410 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587416 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587420 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587424 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587428 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587432 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587436 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587442 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.590947 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587446 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587450 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587454 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587458 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587462 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587466 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587470 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587475 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587479 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587483 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587488 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587492 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587496 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587500 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587504 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587508 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587512 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587516 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587520 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.591427 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587524 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587528 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587532 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587536 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587540 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587545 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587550 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587554 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587559 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587563 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587569 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587576 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.587582 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587712 5043 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587726 5043 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587735 5043 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587741 5043 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587748 5043 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587753 5043 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587759 5043 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:03:38.591909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587765 5043 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587770 5043 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587775 5043 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587781 5043 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587786 5043 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587791 5043 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587795 5043 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587800 5043 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587804 5043 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587809 5043 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587814 5043 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587819 5043 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587823 5043 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587828 5043 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587833 5043 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587838 5043 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587845 5043 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587850 5043 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587856 5043 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587861 5043 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587865 5043 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587870 5043 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587875 5043 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587879 5043 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:03:38.592408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587885 5043 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587891 5043 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587896 5043 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587900 5043 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587905 5043 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587910 5043 flags.go:64] FLAG: --enable-server="true" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587914 5043 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587920 5043 flags.go:64] FLAG: --event-burst="100" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587925 5043 flags.go:64] FLAG: --event-qps="50" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587930 5043 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587935 5043 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587939 5043 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587945 5043 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587950 5043 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587955 5043 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587959 5043 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587964 5043 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587969 5043 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587973 5043 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587978 5043 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587983 5043 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587987 5043 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587992 5043 flags.go:64] FLAG: --feature-gates="" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.587998 5043 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588003 5043 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:03:38.593034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588008 5043 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588013 5043 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588018 5043 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588023 5043 flags.go:64] FLAG: --help="false" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588027 5043 flags.go:64] FLAG: --hostname-override="ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588032 5043 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588037 5043 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588042 5043 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588048 5043 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588054 5043 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588058 5043 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588063 5043 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588080 5043 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588085 5043 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588090 5043 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588095 5043 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588100 5043 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588105 5043 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588109 5043 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588114 5043 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588118 5043 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588123 5043 flags.go:64] FLAG: --lock-file="" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588127 5043 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588132 5043 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:03:38.593874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588136 5043 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588145 5043 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588150 5043 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588154 5043 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588159 5043 flags.go:64] FLAG: --logging-format="text" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588163 5043 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588168 5043 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588173 5043 flags.go:64] FLAG: --manifest-url="" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588177 5043 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588185 5043 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588189 5043 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588195 5043 flags.go:64] FLAG: --max-pods="110" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588200 5043 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588204 5043 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588209 5043 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588213 5043 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588218 5043 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588223 5043 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588228 5043 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588238 5043 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588243 5043 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588247 5043 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588252 5043 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588256 5043 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588261 5043 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:03:38.594478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588265 5043 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588270 5043 flags.go:64] FLAG: --port="10250" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588282 5043 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588287 5043 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e6fe418f2be395f6" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588292 5043 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588296 5043 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588301 5043 flags.go:64] FLAG: --register-node="true" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588306 5043 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588311 5043 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588316 5043 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588321 5043 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588325 5043 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588331 5043 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588335 5043 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588340 5043 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588345 5043 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588349 5043 flags.go:64] FLAG: --runonce="false" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588355 5043 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588360 5043 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588365 5043 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588369 5043 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588379 5043 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588384 5043 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588389 5043 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588394 5043 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588398 5043 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:03:38.595093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588403 5043 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588407 5043 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588412 5043 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588417 5043 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588421 5043 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588426 5043 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588433 5043 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588438 5043 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588443 5043 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588448 5043 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588453 5043 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588457 5043 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588462 5043 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588467 5043 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588474 5043 flags.go:64] FLAG: --v="2" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588481 5043 flags.go:64] FLAG: --version="false" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588487 5043 flags.go:64] FLAG: --vmodule="" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588493 5043 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.588498 5043 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588693 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588698 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588702 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588707 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.595714 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588711 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588716 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588720 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588724 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588730 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588735 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588742 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588748 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588753 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588757 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588762 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588766 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588770 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588775 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588779 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588783 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588787 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588791 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588795 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.596298 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588799 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588803 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588807 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588811 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588818 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588822 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588826 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588830 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588834 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588839 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588843 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588847 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588851 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588855 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588859 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588864 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588868 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588875 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588879 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588883 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.596801 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588888 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588892 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588896 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588900 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588904 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588908 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588912 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588916 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588920 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588924 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588929 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588933 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588937 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588941 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588945 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588949 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588955 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588959 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588963 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588967 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.597363 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588971 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588975 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588980 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588984 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588988 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588992 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.588996 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589000 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589005 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589011 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589016 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589020 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589024 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589028 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589033 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589037 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589041 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589045 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589050 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589054 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.597839 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589058 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589062 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589067 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589084 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589088 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589092 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589097 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589101 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589107 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589111 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589115 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589119 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589123 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589127 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589134 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589139 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589143 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589148 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589152 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.598421 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589180 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589186 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589194 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589198 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.589235 5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589287 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589293 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589298 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589303 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589307 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589311 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589315 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589319 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589323 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589328 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589332 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.598884 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589336 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589340 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589344 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589348 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589353 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589357 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589363 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589367 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589372 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589376 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589380 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589384 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589389 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589393 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589397 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589401 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589405 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589410 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589416 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589420 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.600666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589425 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589429 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589433 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589437 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589441 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589445 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589449 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589453 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589457 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589464 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589470 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589474 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589478 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589482 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589487 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589499 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589504 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589508 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589514 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.601335 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589518 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589522 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589526 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589530 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589534 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589538 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589542 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589546 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589550 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589557 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589562 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589570 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589576 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589584 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589589 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589594 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589598 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589603 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589607 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.601869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589611 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589615 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589619 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589623 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589627 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589631 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589635 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589639 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589644 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589648 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589652 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589656 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589662 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589666 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589670 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589674 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589678 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589683 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589686 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589690 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.602726 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589695 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589701 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589705 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589709 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589715 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589719 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589724 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589728 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589732 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589737 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589740 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589745 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589749 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589753 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589757 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589761 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.589765 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.593782 5043 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.593802 5043 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:03:38.603584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.593817 5043 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.593821 5043 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593876 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593882 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593886 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593889 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593892 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593895 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593898 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593901 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593903 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593906 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593909 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593912 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593914 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593918 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593921 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593923 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593926 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.604194 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593929 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593931 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593934 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593937 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593940 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593943 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593945 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593948 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593951 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593953 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593956 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593959 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593967 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593970 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593973 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593975 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593978 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593980 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593983 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.604747 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593987 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593990 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593993 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593996 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.593999 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594002 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594004 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594008 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594010 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594013 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594015 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594018 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594020 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594023 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594026 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594028 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594031 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594034 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594036 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594039 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.605269 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594041 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594044 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594046 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594049 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594051 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594054 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594064 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594082 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594087 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594089 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594092 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594095 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594097 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594100 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594102 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594105 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594107 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594110 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594113 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.605781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594116 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594120 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594124 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594127 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594130 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594133 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594135 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594138 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594140 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594143 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594145 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594148 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594150 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594153 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594156 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594158 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594160 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594163 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594165 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.606310 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594168 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594178 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594180 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594183 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594186 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594188 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594191 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594193 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594196 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594198 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594201 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594204 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.594237 5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594408 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594413 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594416 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.606776 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594419 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594421 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594424 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594426 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594429 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594432 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594434 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594437 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594439 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594442 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594444 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594447 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594450 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594452 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594455 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594457 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594460 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594462 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594470 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.607292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594473 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594476 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594478 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594481 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594484 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594487 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594489 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594492 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594495 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594498 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594501 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594503 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594506 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594509 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594511 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594513 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594516 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594519 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594521 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.607775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594524 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594526 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594529 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594532 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594534 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594536 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594539 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594541 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594544 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594546 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594549 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594551 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594554 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594562 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594565 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594568 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594570 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594573 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594576 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594578 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.608259 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594581 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594586 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594588 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594591 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594595 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594599 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594603 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594606 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594608 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594611 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594613 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594616 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594619 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594622 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594624 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594627 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594629 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594632 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594634 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.608748 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594637 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594639 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594642 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594644 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594647 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594649 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594652 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594660 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594663 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594666 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594668 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594671 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594673 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594676 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594680 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594683 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594687 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594690 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594693 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594696 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.609285 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594699 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594702 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594704 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594707 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594710 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594712 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.594736 5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594774 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594778 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594781 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594784 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594786 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594789 5043 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594791 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594794 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594796 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:03:38.609785 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594799 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594802 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594805 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594808 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594816 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594819 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594821 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594824 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594826 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594829 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594831 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594837 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594839 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594842 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594844 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594847 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594850 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594853 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594856 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594858 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:03:38.610216 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594861 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594863 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594866 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594869 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594871 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594874 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594876 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594879 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594882 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594884 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594886 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594889 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594892 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594894 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594897 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594900 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594902 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594910 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594913 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594916 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:03:38.610742 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594919 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594921 5043 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594924 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594929 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594931 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594934 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594936 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594939 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594942 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594945 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594949 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594953 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594957 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594961 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594964 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594966 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594969 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594972 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594974 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:03:38.611348 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594977 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594980 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594982 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594985 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594987 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594990 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594992 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594995 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.594997 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595000 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595002 5043 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595010 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595013 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595016 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595018 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595021 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595025 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595027 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595030 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595032 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:03:38.611910 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595035 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595038 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595042 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595045 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595048 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595052 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595055 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595058 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595060 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595063 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595066 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595081 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595085 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595087 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595090 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595093 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595095 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:38.595098 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.595156 5043 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:03:38.612566 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596136 5043 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596185 5043 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596390 5043 server.go:1014] "Starting client certificate rotation" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596526 5043 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596575 5043 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:57:58 +0000 UTC" deadline="2028-07-26 07:53:46.285386772 +0000 UTC" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596604 5043 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15131h50m7.688784224s" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.596884 5043 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.598049 5043 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.601248 5043 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.603443 5043 log.go:25] "Validated CRI v1 image API" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.604253 5043 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:03:38.613190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.606315 5043 fs.go:135] Filesystem UUIDs: map[200ac875-8a5f-406f-a9c6-c9f934e72ba2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89b1c9fd-f13d-4de3-b435-c947ba25d5e1:/dev/nvme0n1p3] Nov 03 20:03:38.613593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.606337 5043 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e2dbcbefeaf5f027961da4c542791e61adb71a3b3ffa4c239d98f6a35617874/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e2dbcbefeaf5f027961da4c542791e61adb71a3b3ffa4c239d98f6a35617874/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55c3754192e88cd9b36ef2b7713e68b41da83840a1cb25380601a5b3c5d5b8e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55c3754192e88cd9b36ef2b7713e68b41da83840a1cb25380601a5b3c5d5b8e9/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ab7430f768e591d2e24ba01ded4ea75e3859ea2c27dc93fe4ad7c89cdebabbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ab7430f768e591d2e24ba01ded4ea75e3859ea2c27dc93fe4ad7c89cdebabbf/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb9b3af502ed6e1dd7485719d8f06c9dd4567f2a7d96b2c4b6e6565197138fbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb9b3af502ed6e1dd7485719d8f06c9dd4567f2a7d96b2c4b6e6565197138fbc/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0138ff1f-c12a-446e-bd06-e62f8202a476/volumes/kubernetes.io~projected/kube-api-access-4sqkl:{mountpoint:/var/lib/kubelet/pods/0138ff1f-c12a-446e-bd06-e62f8202a476/volumes/kubernetes.io~projected/kube-api-access-4sqkl major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0138ff1f-c12a-446e-bd06-e62f8202a476/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0138ff1f-c12a-446e-bd06-e62f8202a476/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a8e9079-e494-4815-a2dc-fc3aa2763628/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0a8e9079-e494-4815-a2dc-fc3aa2763628/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18f94909-ffd1-4bea-ad03-10772d7775f3/volumes/kubernetes.io~projected/kube-api-access-pjk6w:{mountpoint:/var/lib/kubelet/pods/18f94909-ffd1-4bea-ad03-10772d7775f3/volumes/kubernetes.io~projected/kube-api-access-pjk6w major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c49d972-a6cc-488a-90b7-8393e60a93a8/volumes/kubernetes.io~projected/kube-api-access-8m7jx:{mountpoint:/var/lib/kubelet/pods/1c49d972-a6cc-488a-90b7-8393e60a93a8/volumes/kubernetes.io~projected/kube-api-access-8m7jx major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25244768-eb42-47ea-a28d-20609fc12f42/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/25244768-eb42-47ea-a28d-20609fc12f42/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~projected/kube-api-access-p9sbx:{mountpoint:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~projected/kube-api-access-p9sbx major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26f26e13-d1c9-4747-914f-af2fb15d7ed7/volumes/kubernetes.io~projected/kube-api-access-4p8g9:{mountpoint:/var/lib/kubelet/pods/26f26e13-d1c9-4747-914f-af2fb15d7ed7/volumes/kubernetes.io~projected/kube-api-access-4p8g9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b318e41-0c85-4303-addc-336f59f4a4e7/volumes/kubernetes.io~projected/kube-api-access-7h556:{mountpoint:/var/lib/kubelet/pods/2b318e41-0c85-4303-addc-336f59f4a4e7/volumes/kubernetes.io~projected/kube-api-access-7h556 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bd39722-fd14-42d7-af96-2b37b2bba425/volumes/kubernetes.io~projected/kube-api-access-mng2w:{mountpoint:/var/lib/kubelet/pods/2bd39722-fd14-42d7-af96-2b37b2bba425/volumes/kubernetes.io~projected/kube-api-access-mng2w major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31637ee3-ae67-4836-992c-3553f96a85e5/volumes/kubernetes.io~projected/kube-api-access-tl7bf:{mountpoint:/var/lib/kubelet/pods/31637ee3-ae67-4836-992c-3553f96a85e5/volumes/kubernetes.io~projected/kube-api-access-tl7bf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volumes/kubernetes.io~projected/kube-api-access-7vs5k:{mountpoint:/var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volumes/kubernetes.io~projected/kube-api-access-7vs5k major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47ee139e-2eb1-4e93-9536-7c36c003ff40/volumes/kubernetes.io~projected/kube-api-access-g6brv:{mountpoint:/var/lib/kubelet/pods/47ee139e-2eb1-4e93-9536-7c36c003ff40/volumes/kubernetes.io~projected/kube-api-access-g6brv major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57170d5a-5e31-40d9-aebd-645e3c12c610/volumes/kubernetes.io~projected/kube-api-access-vzpbh:{mountpoint:/var/lib/kubelet/pods/57170d5a-5e31-40d9-aebd-645e3c12c610/volumes/kubernetes.io~projected/kube-api-access-vzpbh major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70625344-a07f-4396-8843-f66a9f5421d1/volumes/kubernetes.io~projected/kube-api-access-ddtdg:{mountpoint:/var/lib/kubelet/pods/70625344-a07f-4396-8843-f66a9f5421d1/volumes/kubernetes.io~projected/kube-api-access-ddtdg major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70625344-a07f-4396-8843-f66a9f5421d1/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/70625344-a07f-4396-8843-f66a9f5421d1/volumes/kubernetes.io~secret/signing-key major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/781d2937-5b44-4786-8c19-7629f4548399/volumes/kubernetes.io~projected/kube-api-access-79mk2:{mountpoint:/var/lib/kubelet/pods/781d2937-5b44-4786-8c19-7629f4548399/volumes/kubernetes.io~projected/kube-api-access-79mk2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~projected/kube-api-access-9pzbb:{mountpoint:/var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~projected/kube-api-access-9pzbb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/967ed630-92ed-447b-b106-bb39959cbf0b/volumes/kubernetes.io~projected/kube-api-access-v2l6w:{mountpoint:/var/lib/kubelet/pods/967ed630-92ed-447b-b106-bb39959cbf0b/volumes/kubernetes.io~projected/kube-api-access-v2l6w major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a28d511-b41b-4b0c-8819-6cba4a6620d8/volumes/kubernetes.io~projected/kube-api-access-h4p7d:{mountpoint:/var/lib/kubelet/pods/9a28d511-b41b-4b0c-8819-6cba4a6620d8/volumes/kubernetes.io~projected/kube-api-access-h4p7d major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a28d511-b41b-4b0c-8819-6cba4a6620d8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9a28d511-b41b-4b0c-8819-6cba4a6620d8/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22c4edd-da5c-44d4-a71c-6e162cf7f221/volumes/kubernetes.io~projected/kube-api-access-8t28n:{mountpoint:/var/lib/kubelet/pods/a22c4edd-da5c-44d4-a71c-6e162cf7f221/volumes/kubernetes.io~projected/kube-api-access-8t28n major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7ff3150-0ab6-42d2-81ab-b03e7d944674/volumes/kubernetes.io~projected/kube-api-access-9dt24:{mountpoint:/var/lib/kubelet/pods/a7ff3150-0ab6-42d2-81ab-b03e7d944674/volumes/kubernetes.io~projected/kube-api-access-9dt24 major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa34540b-36c1-4ef4-b4c9-e14a2419ee31/volumes/kubernetes.io~projected/kube-api-access-7f6z7:{mountpoint:/var/lib/kubelet/pods/aa34540b-36c1-4ef4-b4c9-e14a2419ee31/volumes/kubernetes.io~projected/kube-api-access-7f6z7 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa7644a3-9a1a-47a8-82da-44de32fa7ee2/volumes/kubernetes.io~projected/kube-api-access-wm5gq:{mountpoint:/var/lib/kubelet/pods/aa7644a3-9a1a-47a8-82da-44de32fa7ee2/volumes/kubernetes.io~projected/kube-api-access-wm5gq major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~projected/kube-api-access-4nq2q:{mountpoint:/var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~projected/kube-api-access-4nq2q major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~secret/default-certificate major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~secret/stats-auth major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5/volumes/kubernetes.io~projected/kube-api-access-fkq5x:{mountpoint:/var/lib/kubelet/pods/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5/volumes/kubernetes.io~projected/kube-api-access-fkq5x major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5689f3-b955-4d7c-b4b8-c203f7679fa1/volumes/kubernetes.io~projected/kube-api-access-9zw4h:{mountpoint:/var/lib/kubelet/pods/db5689f3-b955-4d7c-b4b8-c203f7679fa1/volumes/kubernetes.io~projected/kube-api-access-9zw4h major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5689f3-b955-4d7c-b4b8-c203f7679fa1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/db5689f3-b955-4d7c-b4b8-c203f7679fa1/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfd74d5b-c18f-42cb-be6d-416d2eb1c921/volumes/kubernetes.io~projected/kube-api-access-k2vrx:{mountpoint:/var/lib/kubelet/pods/dfd74d5b-c18f-42cb-be6d-416d2eb1c921/volumes/kubernetes.io~projected/kube-api-access-k2vrx major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e05b9d70-936b-44f5-b0f2-bc68492d7951/volumes/kubernetes.io~projected/kube-api-access-zhxnd:{mountpoint:/var/lib/kubelet/pods/e05b9d70-936b-44f5-b0f2-bc68492d7951/volumes/kubernetes.io~projected/kube-api-access-zhxnd major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e341cc19-7823-4078-bd1c-1a7d3732882d/volumes/kubernetes.io~projected/kube-api-access-9xtq4:{mountpoint:/var/lib/kubelet/pods/e341cc19-7823-4078-bd1c-1a7d3732882d/volumes/kubernetes.io~projected/kube-api-access-9xtq4 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~projected/kube-api-access-xksm5:{mountpoint:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~projected/kube-api-access-xksm5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed430c0d-f5a9-4cf9-be18-182b648d8814/volumes/kubernetes.io~projected/kube-api-access-442gc:{mountpoint:/var/lib/kubelet/pods/ed430c0d-f5a9-4cf9-be18-182b648d8814/volumes/kubernetes.io~projected/kube-api-access-442gc major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed430c0d-f5a9-4cf9-be18-182b648d8814/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ed430c0d-f5a9-4cf9-be18-182b648d8814/volumes/kubernetes.io~secret/cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edd592ba-dbd2-4026-906b-27478b901db5/volumes/kubernetes.io~projected/kube-api-access-jrt46:{mountpoint:/var/lib/kubelet/pods/edd592ba-dbd2-4026-906b-27478b901db5/volumes/kubernetes.io~projected/kube-api-access-jrt46 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e/volumes/kubernetes.io~projected/kube-api-access-5tb5j:{mountpoint:/var/lib/kubelet/pods/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e/volumes/kubernetes.io~projected/kube-api-access-5tb5j major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fff6f118-0ff9-4478-b759-04e2d3f63420/volumes/kubernetes.io~projected/kube-api-access-cv4ms:{mountpoint:/var/lib/kubelet/pods/fff6f118-0ff9-4478-b759-04e2d3f63420/volumes/kubernetes.io~projected/kube-api-access-cv4ms major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/43fc68990d22246bdd5192ac7ff0432e7704381f73e9fe281e3fc7261541c14a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b52f9dfa86fae669ad5d07a83ddc6706d22ffb9f0f46057557918786be5e0a9c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fd45ff06402429a832437a26bdea03fd41969ba53f67fd17983910430bf58029/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/27715c45e57c03497811e9e5fa4b47ff796e0f2c4143a99e28be3e38777e22be/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bb6707a48b125cc6159a50fb1e0219edc0b4de84816f8e6b5089bf49933873fa/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/49180781a14fdb786a63f078574a1e68c12227e9e21a707de238d6aa7bbe36ba/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c5b1605372f8ee746fa007b3922720a861f22197af4283373be1adf5fb02b806/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f7667f48f282f87432564ddc9ba25df55387e7923c414bd2c1dcfac51b8786c6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3939da2415cd0f8e1e35ded6d1273ff009d45739dde62f128fd7f06b6ef3a515/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/49673f0e9796d701c0d52ad525de56d8770d01d0b2261504508418e416708171/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/51f9097038dda3f7aa6be21a5fd3ca7ba64ff58987e7182ea22d6047377f9224/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/990d659746bfae7dc57591a0f1e17609f82e198d1e9755da6163b833829a9111/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9579a756449127dac03fc8a81c69c1da45cb523f63b45d729dd8f02e154b5ae6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/decc07e982b1e8c301480249eae6edc319abfcaa9c8a89001415a0f3485dfa9f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/57efade35e59510db8936f65f5d163ebe3a4d432955938ad6cc4e583eb8a6337/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d68fad7154e30e80996e7acf70e6b9c58cd48631bb823e73b8fe7d6b2e7ac385/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6f9c4bb5dc5c19c8fa90703bf5d7527909594b732b3ea2f89ddcc5234f4dd445/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/70e6320f9b14459a5c5e4499f4754bd6b57a377cb0b6e229faf77fd827ab692b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/71f5f293b644fa100b43fb44684c171fb73fcb5a69648d1cd56283317310577c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9c50b7f097c04945b5d3a99590733831e4347dfb4dc22933526f713d3f6877f9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2fb299c7664f6df825ada8a372e7bcd6042469e309d1207f0e0890ecf4eb280d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6f00398c0b675106dcb775d5a7c4318c0347bd104f8a302318d38bf0d4494ec6/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/509d62ea147ed2f0c533e434f3e29191a349ec260acc0c996803ccf7151bac3b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8dcc5926ee4a5b0454c0e0bfa8d2de60e9bc4f298d5e8cbe717cd719b38525d4/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/34229d94cc44a96a77a9204f338efd35b76770135de154f03fe4864b79a4aa50/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c77f9e543b426742f93e90817b7f22096fd2e1a2dd5bef6d21f0de27a76157e9/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/9bd182ced9a95a1434403cfb54f1fc13170b1a6e375dca2123d10b2efdcaabe1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d4512301ab789bf453dad40811741a2bcacb46e0f92455fe1550d9b8fa86dad5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/23c6cf883d17f4c4cea60eaa8b3e65537eec25d67a6cb88b494fa82d4b2ce993/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/d4d3b965c45402efd20e3b62db6e2fec350633a7b4094a226afaa87350d7ee53/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6ceda5521ac0a1b6620cbdb96c90b18120fe260c7f6719c29059fc629f896e16/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/0fba909930cb3529f9d0389f4c5084d19340f1f07a1469fcb240994e5f5b6f2a/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/58d6dbb85ec682a1313218c09fbc8820158b239c8be1d8c42272ff0922162340/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/5c8dbec2dd99176109dfa18c16fe948960ba8a50f7d5702909d3e955c8043cc8/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/2968db67785285294a41a1636bf9039129ee06be3eca3f122aca80c1a3553bc5/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e49e72f0a2337ee63ffbfbc3761ca780e59dd3720bc271a00911077291e115b2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/b91c7c2b95fd29c7992d6eee554a8f5c320cb0d3a389ba5a20f8c2f4efff9fe6/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/eb2d38ccf4faff829e9f243f2b348d251b90a4c0f2b3ef66b6b6314933dbc29d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a7da9bffd945963912c9a3bd727b67738442dff12203c20f0533b5de05e59086/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/b69ca9cc1e3591dd2092267a16004d13c2879b5ba85f3c271c1ccae9da25d161/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/8b2551d76abd6784220d7acb98ec562af33f72b95b3a73207429fbd35009971f/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/68830ca42b0b045507db24cd236828cd51ebb4c7113dcd094c862affb6dfb059/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/0a95080bdc9d26072597465127de4d398c58aa4a82ee16f61578f7a5f64b5e8e/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/bb47a777f0ae13eec10f1ed2ef32af1df5db0b5051c829f99b8970969aa4e9df/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/71042ddd26aba79c176fa447665f9ee50d1ea4d92d21ba11ac0df08d054fc370/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/658f4b2dd7b38fd9628ec5673530f396a126590fdf02bcae0ce0e921bc099f70/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/48b88986c512a69801360cb1511da287ec748f2877f2037285674cbc6f01bf5a/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/4a0108197018ae9a48a63161ce45adc5dec4aac5bfabb5aeb5efff6e7efa41f8/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/b5c44df49b3bc42cc99e184027bb063b53c0263fce36d73e70e14246cb289bdc/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/f6d5d7d37e56f038d6d8f47a9321ffe6615928b23df8a65e1f4bcb9e258c5674/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/a0b85617f00cc601c801674eb3715f742a752bae1dfb5bc870e2e46ea0bf0fd6/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/346c03060a08d12e37efeb3816c57d8206350ca33d0db6c33393083c537a70e4/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/6467801b12c390a7c1a5c2fe480c283c8302d824796437a0ba48d20487be8ec2/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9cd9445bbbbe5892ab85edcd962b48c04f0e041b8670886c3565350675a88958/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/13431e921a6676ba9cc85a79a1f9a9882da0d6528174452911d95331cae88854/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/6e8cb671f045c51682659d7cfb776dd2a75cec0b454649b8ff0b543ae28bdc97/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/05fc7834bfa75ec296df54dc9f881dff4791ace593ff1901e57e7e9d80438642/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/bb4630dfc45b85eb3dcdae5efc9e36b92b9f5521c89e037bb218da3a5868ff7d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/84ec0da974dc281af500c94dcc0e9fff136936996a3a44cf7db8cd9a9745e39e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3a8d75c53def2947effd304d955d8f80ed0e4b1ac31adac828912db3d53bc8c3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c9c3bfbfcd84dd0ff7a4157d108334e037c540ecefb9edbca8f30c6429f9c379/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8e97584a9740f2d536c3fdb2edb92d15a9c7fd9a781e2babac8f0d06467a8444/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 20:03:38.614715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.614350 5043 manager.go:217] Machine: {Timestamp:2026-11-03 20:03:38.613878349 +0000 UTC m=+0.080833594 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099621 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279b67898c945b792fe3c398371621 SystemUUID:ec279b67-898c-945b-792f-e3c398371621 BootID:a7194bf7-4d81-43ca-9bc0-747d953ab879 Filesystems:[{Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c49d972-a6cc-488a-90b7-8393e60a93a8/volumes/kubernetes.io~projected/kube-api-access-8m7jx DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0138ff1f-c12a-446e-bd06-e62f8202a476/volumes/kubernetes.io~projected/kube-api-access-4sqkl DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/781d2937-5b44-4786-8c19-7629f4548399/volumes/kubernetes.io~projected/kube-api-access-79mk2 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a28d511-b41b-4b0c-8819-6cba4a6620d8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa7644a3-9a1a-47a8-82da-44de32fa7ee2/volumes/kubernetes.io~projected/kube-api-access-wm5gq DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47ee139e-2eb1-4e93-9536-7c36c003ff40/volumes/kubernetes.io~projected/kube-api-access-g6brv DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2bd39722-fd14-42d7-af96-2b37b2bba425/volumes/kubernetes.io~projected/kube-api-access-mng2w DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fff6f118-0ff9-4478-b759-04e2d3f63420/volumes/kubernetes.io~projected/kube-api-access-cv4ms DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80f861b61cc2657f78e62260a951f56e2170dc76e4c1039da0ea50ab4eae20ed/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/70625344-a07f-4396-8843-f66a9f5421d1/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31637ee3-ae67-4836-992c-3553f96a85e5/volumes/kubernetes.io~projected/kube-api-access-tl7bf DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bd42b62ae6e38968a5d57cc05d355adbb17de8adc99e75cba06ec38d289454f/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07f953a183ea9eae36ba3aafde854501166f43e9938ea75f13136622525a397c/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/18f94909-ffd1-4bea-ad03-10772d7775f3/volumes/kubernetes.io~projected/kube-api-access-pjk6w DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26f26e13-d1c9-4747-914f-af2fb15d7ed7/volumes/kubernetes.io~projected/kube-api-access-4p8g9 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25244768-eb42-47ea-a28d-20609fc12f42/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~projected/kube-api-access-9pzbb DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaf049de0eb5966e9f51f28fc4ad7fa6df7ec10ee9f63ea2f9ff8882b0d5e3d8/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0138ff1f-c12a-446e-bd06-e62f8202a476/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~projected/kube-api-access-xksm5 DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55c3754192e88cd9b36ef2b7713e68b41da83840a1cb25380601a5b3c5d5b8e9/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/954386bc5a98ff1c7a78dcc63d818c5a357d19f4f5efe10a1c2a2306bfa559fe/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63ecdaf6a470c46f72e7e11833a6092971cb8a20fe17d5c1ab0f727a6dc0afcf/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e/volumes/kubernetes.io~projected/kube-api-access-5tb5j DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb9b3af502ed6e1dd7485719d8f06c9dd4567f2a7d96b2c4b6e6565197138fbc/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db5689f3-b955-4d7c-b4b8-c203f7679fa1/volumes/kubernetes.io~projected/kube-api-access-9zw4h DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a22c4edd-da5c-44d4-a71c-6e162cf7f221/volumes/kubernetes.io~projected/kube-api-access-8t28n DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6be77432ddaa310043148fccb8e28ad6026d183eab6af9701a437aefbf3a6bde/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e2dbcbefeaf5f027961da4c542791e61adb71a3b3ffa4c239d98f6a35617874/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eeee13e1b45d27bc312794d84c059e8a756cba525ea0db9ccec57a1e5aa5c660/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/57170d5a-5e31-40d9-aebd-645e3c12c610/volumes/kubernetes.io~projected/kube-api-access-vzpbh DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed430c0d-f5a9-4cf9-be18-182b648d8814/volumes/kubernetes.io~projected/kube-api-access-442gc DeviceMajor:0 DeviceMinor:243 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ac7580fa1499080e665d5d8244e78b2af50f6d14750c39c27ae3bdcfbac0097/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e341cc19-7823-4078-bd1c-1a7d3732882d/volumes/kubernetes.io~projected/kube-api-access-9xtq4 DeviceMajor:0 DeviceMinor:334 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66b4ffa1dfa45524f4d172cacbd263122ef00a5d9d8be4c3d3349170735fed2b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07f96399f36e4f10bdaa9df8348e2179f1216391b029fd0530a668d8646a2b39/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12f2424da3a21d97a0c3ce515773b521bdd7df5f2834fac31d48ea9d268b43b2/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/70625344-a07f-4396-8843-f66a9f5421d1/volumes/kubernetes.io~projected/kube-api-access-ddtdg DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa34540b-36c1-4ef4-b4c9-e14a2419ee31/volumes/kubernetes.io~projected/kube-api-access-7f6z7 DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8316b7fdd50109e35eb9b7bc4098031bbb4ce3acd782566483c48483fe64ed3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14a06a20b5b0f655d05ce552741c164dffae9ac2495eabc48d58073853174bc8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9a28d511-b41b-4b0c-8819-6cba4a6620d8/volumes/kubernetes.io~projected/kube-api-access-h4p7d DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd59109d8683a957968f503b16e25eacabfc603181652c819da4178cc8080356/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ed430c0d-f5a9-4cf9-be18-182b648d8814/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90f3e0d1f495edbd2b5e651c6c64cb9d28c87f9c70786aed46818564494e2170/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5/volumes/kubernetes.io~projected/kube-api-access-fkq5x DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c2062bc3-f9db-48f9-aedb-b1367774ee26/volumes/kubernetes.io~projected/kube-api-access-4nq2q DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ab7430f768e591d2e24ba01ded4ea75e3859ea2c27dc93fe4ad7c89cdebabbf/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db5689f3-b955-4d7c-b4b8-c203f7679fa1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~projected/kube-api-access-p9sbx DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/967ed630-92ed-447b-b106-bb39959cbf0b/volumes/kubernetes.io~projected/kube-api-access-v2l6w DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/331068c4-2a21-4ab6-9fbf-ea1ce605d767/volumes/kubernetes.io~projected/kube-api-access-7vs5k DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b318e41-0c85-4303-addc-336f59f4a4e7/volumes/kubernetes.io~projected/kube-api-access-7h556 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60fb98393bfd471a33a7e5e7605c3b07498dcfbc67096ae8eab3eb6a6f1954f7/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ecf92d9-64ba-4348-9124-5f02d8032c9c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0a8e9079-e494-4815-a2dc-fc3aa2763628/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/edd592ba-dbd2-4026-906b-27478b901db5/volumes/kubernetes.io~projected/kube-api-access-jrt46 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2df1e49c5ed2942d52db0fa4be58a53bc6e11738efe5f13a4496e6b74d42b8b2/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a7ff3150-0ab6-42d2-81ab-b03e7d944674/volumes/kubernetes.io~projected/kube-api-access-9dt24 DeviceMajor:0 DeviceMinor:238 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dfd74d5b-c18f-42cb-be6d-416d2eb1c921/volumes/kubernetes.io~projected/kube-api-access-k2vrx DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e05b9d70-936b-44f5-b0f2-bc68492d7951/volumes/kubernetes.io~projected/kube-api-access-zhxnd DeviceMajor:0 DeviceMinor:242 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51db481c4a766041200d21f32459136c063d36838a103b7f9591da668154193c/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88d3d3c32a4fa29ab66d44f1821bfcf98090e8376e31d27eca26cf267be2df16/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07f953a183ea9ea MacAddress:06:ee:f6:a7:82:21 Speed:10000 Mtu:8901} {Name:0e2dbcbefeaf5f0 MacAddress:06:ad:91:30:6f:83 Speed:10000 Mtu:8901} {Name:12f2424da3a21d9 MacAddress:fe:de:aa:35:c1:c4 Speed:10000 Mtu:8901} {Name:2df1e49c5ed2942 MacAddress:0e:1a:ab:9d:f7:29 Speed:10000 Mtu:8901} {Name:51db481c4a76604 MacAddress:d2:cd:da:a0:71:73 Speed:10000 Mtu:8901} {Name:55c3754192e88cd MacAddress:1a:fc:4f:5b:f7:28 Speed:10000 Mtu:8901} {Name:60fb98393bfd471 MacAddress:8a:3c:21:0d:c4:58 Speed:10000 Mtu:8901} {Name:63ecdaf6a470c46 MacAddress:72:56:b0:63:6f:c9 Speed:10000 Mtu:8901} {Name:6ac7580fa149908 MacAddress:d2:80:8b:48:a2:55 Speed:10000 Mtu:8901} {Name:6be77432ddaa310 MacAddress:b6:50:f4:ea:cc:e2 Speed:10000 Mtu:8901} {Name:80f861b61cc2657 MacAddress:b2:b3:ec:22:f2:9c Speed:10000 Mtu:8901} {Name:88d3d3c32a4fa29 MacAddress:86:89:00:ce:ec:09 Speed:10000 Mtu:8901} {Name:90f3e0d1f495edb MacAddress:c2:7b:ef:45:1a:d7 Speed:10000 Mtu:8901} {Name:9ab7430f768e591 MacAddress:3a:3e:aa:36:1e:dd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:f3:60:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:72:9a:ac:f8:c2:3f Speed:0 Mtu:8901} {Name:dd59109d8683a95 MacAddress:76:c5:d2:3c:ce:b4 Speed:10000 Mtu:8901} {Name:eb9b3af502ed6e1 MacAddress:76:1b:61:08:37:82 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:f3:60:49 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:52:7a:bb:0d:a1:2b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:90:41:c9:4e:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:03:38.614951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.614720 5043 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:03:38.614951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.614760 5043 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:03:38.615062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615039 5043 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:03:38.615223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615064 5043 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:03:38.615271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615227 5043 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:03:38.615271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615236 5043 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:03:38.615271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615250 5043 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:03:38.615271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615267 5043 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:03:38.615383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615286 5043 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:03:38.615473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615463 5043 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:03:38.615571 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615563 5043 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:03:38.615610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615596 5043 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:03:38.615610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615606 5043 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:03:38.615672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615615 5043 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:03:38.615672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615623 5043 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:03:38.615672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.615630 5043 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:03:38.616553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.616537 5043 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:03:38.616553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.616550 5043 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:03:38.616967 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.616951 5043 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:03:38.620965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.620949 5043 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:03:38.620965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.620963 5043 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:03:38.621186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621063 5043 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:03:38.621275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621264 5043 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:03:38.621307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621294 5043 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:03:38.621398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621389 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:03:38.621434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621405 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:03:38.621434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621411 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:03:38.621434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621417 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:03:38.621434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621424 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:03:38.621434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621433 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621444 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621452 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621460 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621466 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621474 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621484 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621508 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:03:38.621563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.621517 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:03:38.625018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.625003 5043 server.go:1263] "Started kubelet" Nov 03 20:03:38.626286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.625247 5043 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:03:38.626286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.625838 5043 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:03:38.626286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.625899 5043 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:03:38.626286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.626167 5043 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:03:38.626517 ip-10-0-140-102 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:03:38.626680 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.626664 5043 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.629542 5043 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.629634 5043 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.629660 5043 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.629707 5043 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.629715 5043 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.630288 5043 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.630802 5043 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:57:59 +0000 UTC" deadline="2028-06-27 10:24:59.84793189 +0000 UTC" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.630835 5043 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14438h21m21.217100031s" Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634202 5043 factory.go:55] Registering systemd factory Nov 03 20:03:38.634291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634234 5043 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:03:38.634827 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634789 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="781d2937-5b44-4786-8c19-7629f4548399" volumeName="kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-cni-binary-copy" seLinuxMountContext="" Nov 03 20:03:38.634870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634832 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a28d511-b41b-4b0c-8819-6cba4a6620d8" volumeName="kubernetes.io/projected/9a28d511-b41b-4b0c-8819-6cba4a6620d8-kube-api-access-h4p7d" seLinuxMountContext="" Nov 03 20:03:38.634870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634846 5043 factory.go:153] Registering CRI-O factory Nov 03 20:03:38.634870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634861 5043 factory.go:223] Registration of the crio container factory successfully Nov 03 20:03:38.634997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634941 5043 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:03:38.634997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634970 5043 factory.go:103] Registering Raw factory Nov 03 20:03:38.634997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634984 5043 manager.go:1199] Started watching for new ooms in manager Nov 03 20:03:38.635437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635424 5043 manager.go:319] Starting recovery of all containers Nov 03 20:03:38.635741 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.634846 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" volumeName="kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:03:38.635853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635755 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" volumeName="kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle" seLinuxMountContext="" Nov 03 20:03:38.635853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635779 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2062bc3-f9db-48f9-aedb-b1367774ee26" volumeName="kubernetes.io/projected/c2062bc3-f9db-48f9-aedb-b1367774ee26-kube-api-access-4nq2q" seLinuxMountContext="" Nov 03 20:03:38.635853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635799 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5689f3-b955-4d7c-b4b8-c203f7679fa1" volumeName="kubernetes.io/configmap/db5689f3-b955-4d7c-b4b8-c203f7679fa1-config" seLinuxMountContext="" Nov 03 20:03:38.635853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635815 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:03:38.635853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635831 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c49d972-a6cc-488a-90b7-8393e60a93a8" volumeName="kubernetes.io/configmap/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-config" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635853 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ee139e-2eb1-4e93-9536-7c36c003ff40" volumeName="kubernetes.io/projected/47ee139e-2eb1-4e93-9536-7c36c003ff40-kube-api-access-g6brv" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635873 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70625344-a07f-4396-8843-f66a9f5421d1" volumeName="kubernetes.io/secret/70625344-a07f-4396-8843-f66a9f5421d1-signing-key" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635896 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" volumeName="kubernetes.io/secret/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-serving-cert" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635912 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635932 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed430c0d-f5a9-4cf9-be18-182b648d8814" volumeName="kubernetes.io/projected/ed430c0d-f5a9-4cf9-be18-182b648d8814-kube-api-access-442gc" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635948 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed430c0d-f5a9-4cf9-be18-182b648d8814" volumeName="kubernetes.io/secret/ed430c0d-f5a9-4cf9-be18-182b648d8814-cert" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635973 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edd592ba-dbd2-4026-906b-27478b901db5" volumeName="kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-binary-copy" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.635988 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636006 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331068c4-2a21-4ab6-9fbf-ea1ce605d767" volumeName="kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636018 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a28d511-b41b-4b0c-8819-6cba4a6620d8" volumeName="kubernetes.io/secret/9a28d511-b41b-4b0c-8819-6cba4a6620d8-serving-cert" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636033 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7ff3150-0ab6-42d2-81ab-b03e7d944674" volumeName="kubernetes.io/projected/a7ff3150-0ab6-42d2-81ab-b03e7d944674-kube-api-access-9dt24" seLinuxMountContext="" Nov 03 20:03:38.636101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636066 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfd74d5b-c18f-42cb-be6d-416d2eb1c921" volumeName="kubernetes.io/projected/dfd74d5b-c18f-42cb-be6d-416d2eb1c921-kube-api-access-k2vrx" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636110 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636128 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331068c4-2a21-4ab6-9fbf-ea1ce605d767" volumeName="kubernetes.io/projected/331068c4-2a21-4ab6-9fbf-ea1ce605d767-kube-api-access-7vs5k" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636142 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22c4edd-da5c-44d4-a71c-6e162cf7f221" volumeName="kubernetes.io/projected/a22c4edd-da5c-44d4-a71c-6e162cf7f221-kube-api-access-8t28n" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636154 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e05b9d70-936b-44f5-b0f2-bc68492d7951" volumeName="kubernetes.io/projected/e05b9d70-936b-44f5-b0f2-bc68492d7951-kube-api-access-zhxnd" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636173 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636184 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636202 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff6f118-0ff9-4478-b759-04e2d3f63420" volumeName="kubernetes.io/configmap/fff6f118-0ff9-4478-b759-04e2d3f63420-serviceca" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636216 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff6f118-0ff9-4478-b759-04e2d3f63420" volumeName="kubernetes.io/projected/fff6f118-0ff9-4478-b759-04e2d3f63420-kube-api-access-cv4ms" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636245 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0138ff1f-c12a-446e-bd06-e62f8202a476" volumeName="kubernetes.io/projected/0138ff1f-c12a-446e-bd06-e62f8202a476-kube-api-access-4sqkl" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636257 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a8e9079-e494-4815-a2dc-fc3aa2763628" volumeName="kubernetes.io/configmap/0a8e9079-e494-4815-a2dc-fc3aa2763628-konnectivity-ca" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636281 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18f94909-ffd1-4bea-ad03-10772d7775f3" volumeName="kubernetes.io/projected/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-api-access-pjk6w" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636294 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636308 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331068c4-2a21-4ab6-9fbf-ea1ce605d767" volumeName="kubernetes.io/secret/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636328 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85530c5c-c1ec-4d44-8a76-1ea11f34c7a2" volumeName="kubernetes.io/configmap/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-nginx-conf" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636340 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ecf92d9-64ba-4348-9124-5f02d8032c9c" volumeName="kubernetes.io/projected/8ecf92d9-64ba-4348-9124-5f02d8032c9c-kube-api-access-9pzbb" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636369 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" volumeName="kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636382 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd39722-fd14-42d7-af96-2b37b2bba425" volumeName="kubernetes.io/configmap/2bd39722-fd14-42d7-af96-2b37b2bba425-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636395 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331068c4-2a21-4ab6-9fbf-ea1ce605d767" volumeName="kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-config" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636413 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" volumeName="kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-service-ca-bundle" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636431 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" volumeName="kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-tmp" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636450 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5689f3-b955-4d7c-b4b8-c203f7679fa1" volumeName="kubernetes.io/secret/db5689f3-b955-4d7c-b4b8-c203f7679fa1-serving-cert" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636464 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edd592ba-dbd2-4026-906b-27478b901db5" volumeName="kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636478 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa0036d1-fc35-4afe-9c9c-f252b3c34b3e" volumeName="kubernetes.io/projected/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e-kube-api-access-5tb5j" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636496 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0138ff1f-c12a-446e-bd06-e62f8202a476" volumeName="kubernetes.io/secret/0138ff1f-c12a-446e-bd06-e62f8202a476-serving-cert" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636510 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18f94909-ffd1-4bea-ad03-10772d7775f3" volumeName="kubernetes.io/empty-dir/18f94909-ffd1-4bea-ad03-10772d7775f3-data-volume" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636529 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="781d2937-5b44-4786-8c19-7629f4548399" volumeName="kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-multus-daemon-config" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636544 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2062bc3-f9db-48f9-aedb-b1367774ee26" volumeName="kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-stats-auth" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636558 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7ff3150-0ab6-42d2-81ab-b03e7d944674" volumeName="kubernetes.io/empty-dir/a7ff3150-0ab6-42d2-81ab-b03e7d944674-tmp-dir" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636580 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e05b9d70-936b-44f5-b0f2-bc68492d7951" volumeName="kubernetes.io/configmap/e05b9d70-936b-44f5-b0f2-bc68492d7951-telemetry-config" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636594 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f26e13-d1c9-4747-914f-af2fb15d7ed7" volumeName="kubernetes.io/projected/26f26e13-d1c9-4747-914f-af2fb15d7ed7-kube-api-access-4p8g9" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636623 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31637ee3-ae67-4836-992c-3553f96a85e5" volumeName="kubernetes.io/projected/31637ee3-ae67-4836-992c-3553f96a85e5-kube-api-access-tl7bf" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636637 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5689f3-b955-4d7c-b4b8-c203f7679fa1" volumeName="kubernetes.io/projected/db5689f3-b955-4d7c-b4b8-c203f7679fa1-kube-api-access-9zw4h" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636654 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b318e41-0c85-4303-addc-336f59f4a4e7" volumeName="kubernetes.io/projected/2b318e41-0c85-4303-addc-336f59f4a4e7-kube-api-access-7h556" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636671 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57170d5a-5e31-40d9-aebd-645e3c12c610" volumeName="kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh" seLinuxMountContext="" Nov 03 20:03:38.636661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636684 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" volumeName="kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636703 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2062bc3-f9db-48f9-aedb-b1367774ee26" volumeName="kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-default-certificate" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636739 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" volumeName="kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636751 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636767 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0138ff1f-c12a-446e-bd06-e62f8202a476" volumeName="kubernetes.io/configmap/0138ff1f-c12a-446e-bd06-e62f8202a476-config" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636778 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636793 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f26e13-d1c9-4747-914f-af2fb15d7ed7" volumeName="kubernetes.io/empty-dir/26f26e13-d1c9-4747-914f-af2fb15d7ed7-tmp-dir" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636803 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" volumeName="kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636814 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edd592ba-dbd2-4026-906b-27478b901db5" volumeName="kubernetes.io/projected/edd592ba-dbd2-4026-906b-27478b901db5-kube-api-access-jrt46" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636832 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70625344-a07f-4396-8843-f66a9f5421d1" volumeName="kubernetes.io/projected/70625344-a07f-4396-8843-f66a9f5421d1-kube-api-access-ddtdg" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636843 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ecf92d9-64ba-4348-9124-5f02d8032c9c" volumeName="kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-tuned" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636857 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a28d511-b41b-4b0c-8819-6cba4a6620d8" volumeName="kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-config" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636868 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa34540b-36c1-4ef4-b4c9-e14a2419ee31" volumeName="kubernetes.io/projected/aa34540b-36c1-4ef4-b4c9-e14a2419ee31-kube-api-access-7f6z7" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636878 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edd592ba-dbd2-4026-906b-27478b901db5" volumeName="kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636894 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636906 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a8e9079-e494-4815-a2dc-fc3aa2763628" volumeName="kubernetes.io/secret/0a8e9079-e494-4815-a2dc-fc3aa2763628-agent-certs" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636933 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25244768-eb42-47ea-a28d-20609fc12f42" volumeName="kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636944 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636961 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="331068c4-2a21-4ab6-9fbf-ea1ce605d767" volumeName="kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-env-overrides" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636978 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c49d972-a6cc-488a-90b7-8393e60a93a8" volumeName="kubernetes.io/projected/1c49d972-a6cc-488a-90b7-8393e60a93a8-kube-api-access-8m7jx" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.636991 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637009 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd39722-fd14-42d7-af96-2b37b2bba425" volumeName="kubernetes.io/projected/2bd39722-fd14-42d7-af96-2b37b2bba425-kube-api-access-mng2w" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637021 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="781d2937-5b44-4786-8c19-7629f4548399" volumeName="kubernetes.io/projected/781d2937-5b44-4786-8c19-7629f4548399-kube-api-access-79mk2" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637032 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="967ed630-92ed-447b-b106-bb39959cbf0b" volumeName="kubernetes.io/projected/967ed630-92ed-447b-b106-bb39959cbf0b-kube-api-access-v2l6w" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637047 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a28d511-b41b-4b0c-8819-6cba4a6620d8" volumeName="kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-trusted-ca" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637059 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259382b1-dd55-47ac-9744-c5b956a13ce7" volumeName="kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637090 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70625344-a07f-4396-8843-f66a9f5421d1" volumeName="kubernetes.io/configmap/70625344-a07f-4396-8843-f66a9f5421d1-signing-cabundle" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637105 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ecf92d9-64ba-4348-9124-5f02d8032c9c" volumeName="kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-tmp" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637116 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7ff3150-0ab6-42d2-81ab-b03e7d944674" volumeName="kubernetes.io/configmap/a7ff3150-0ab6-42d2-81ab-b03e7d944674-config-volume" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637133 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" volumeName="kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-snapshots" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637145 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" volumeName="kubernetes.io/projected/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-kube-api-access-fkq5x" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637160 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e341cc19-7823-4078-bd1c-1a7d3732882d" volumeName="kubernetes.io/projected/e341cc19-7823-4078-bd1c-1a7d3732882d-kube-api-access-9xtq4" seLinuxMountContext="" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637171 5043 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:03:38.638267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.637178 5043 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:03:38.664176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.664149 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/1.log" Nov 03 20:03:38.664324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.664190 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/1.log" Nov 03 20:03:38.667153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.666086 5043 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:03:38.668812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.668793 5043 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:03:38.668916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.668815 5043 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:03:38.668916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.668837 5043 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:03:38.668916 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.668878 5043 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:03:38.675036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.674988 5043 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:03:38.675597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.675576 5043 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:03:38.681207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.681186 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:38.683526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.683494 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:38.689110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.688794 5043 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:03:38.689110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.688912 5043 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:03:38.704356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.704331 5043 generic.go:356] "Generic (PLEG): container finished" podID="59661663947a8edd85fd5fa4c740ff44" containerID="d9afc4f07db171013d0ad9a79e71dd753566e5f6afc2ddc0397b92eaa5b1e275" exitCode=0 Nov 03 20:03:38.715239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.715218 5043 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="99605c645ed759a56daf436dabdcc40f5748c5172e295b7e29922a79c8eaa781" exitCode=0 Nov 03 20:03:38.715239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.715237 5043 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="a76944204826167b2fab6e5bf6920c0be3acb63d9db1bcda19ad6bc408aafc5b" exitCode=0 Nov 03 20:03:38.715350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.715244 5043 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="23762dceb0f91f695644836c0466a2da19f659336078a3e017e257998354aace" exitCode=0 Nov 03 20:03:38.715350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.715251 5043 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="ac9c900b4479637118deefab19dc27f889daa1150979bb75219da8ed16b86b92" exitCode=0 Nov 03 20:03:38.715350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.715257 5043 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="34e8aa2323d53ef1700e4a08d936a8eb3de6615c306aca9e16c57afda4261a4d" exitCode=0 Nov 03 20:03:38.715350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.715262 5043 generic.go:356] "Generic (PLEG): container finished" podID="edd592ba-dbd2-4026-906b-27478b901db5" containerID="6aa648fe65e92ebc5cbc08a0194656931b1021e5cbb8f61f29361615109b512a" exitCode=0 Nov 03 20:03:38.720496 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.720474 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:03:38.720861 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.720843 5043 generic.go:356] "Generic (PLEG): container finished" podID="331068c4-2a21-4ab6-9fbf-ea1ce605d767" containerID="56729234d733d9bcc35d964ef7d6d73fb3e93d6b9858ea7820a96bf580eedb8a" exitCode=1 Nov 03 20:03:38.729747 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.729726 5043 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-102.ec2.internal\" not found" Nov 03 20:03:38.730632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.730617 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/1.log" Nov 03 20:03:38.730697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.730643 5043 generic.go:356] "Generic (PLEG): container finished" podID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" exitCode=255 Nov 03 20:03:38.746249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.746232 5043 manager.go:324] Recovery completed Nov 03 20:03:38.752706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.752688 5043 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:03:38.767083 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767056 5043 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:03:38.767083 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767084 5043 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:03:38.767205 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767104 5043 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:03:38.767264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767245 5043 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:03:38.767692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767265 5043 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:03:38.767722 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767698 5043 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:03:38.767722 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767705 5043 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:03:38.767722 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767712 5043 policy_none.go:50] "Start" Nov 03 20:03:38.767796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767730 5043 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:03:38.767796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767746 5043 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:03:38.767892 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767882 5043 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:03:38.767923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767892 5043 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:03:38.767923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.767899 5043 policy_none.go:44] "Start" Nov 03 20:03:38.769110 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.769093 5043 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 20:03:38.772309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.772285 5043 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:03:38.774834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.774822 5043 manager.go:341] "Starting Device Plugin manager" Nov 03 20:03:38.774920 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.774845 5043 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:03:38.774920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.774853 5043 server.go:87] "Starting device plugin registration server" Nov 03 20:03:38.775123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775109 5043 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:03:38.775168 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775126 5043 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:03:38.775235 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775218 5043 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:03:38.775353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775321 5043 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:03:38.775471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775363 5043 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:03:38.775471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775369 5043 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:03:38.775573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.775410 5043 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:03:38.775351763Z","UUID":"b60526d5-85f8-4a0d-9eb0-1d9417088b6f","Handler":null,"Name":"","Endpoint":""} Nov 03 20:03:38.776667 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.776646 5043 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:03:38.776887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.776871 5043 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:03:38.776956 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.776898 5043 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:03:38.875621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.875594 5043 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:03:38.877373 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.877357 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientMemory" Nov 03 20:03:38.877452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.877387 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 20:03:38.877452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.877401 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-102.ec2.internal" event="NodeHasSufficientPID" Nov 03 20:03:38.877525 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.877453 5043 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.909213 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.909193 5043 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.909282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.909247 5043 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.969372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969311 5043 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal"] Nov 03 20:03:38.969913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969876 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" event={"ID":"880c789d3d697e88c74ab8f8cf26d9d3","Type":"ContainerStarted","Data":"6070ec032b3df948b320aaa5fcce74c667b82ff7b067067832b70584fb916db4"} Nov 03 20:03:38.969992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969914 5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.969992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969922 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" event={"ID":"880c789d3d697e88c74ab8f8cf26d9d3","Type":"ContainerStarted","Data":"ab5b50ba8205391c0490e31bc90ec41d05dc71bfa579f284ef2d9edc6ed2a5b0"} Nov 03 20:03:38.969992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969956 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" event={"ID":"59661663947a8edd85fd5fa4c740ff44","Type":"ContainerStarted","Data":"6da68763abe3c3a5eb518c5b8470a1c76db47f6711ba3de4b03695c46a94a299"} Nov 03 20:03:38.969992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969974 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" event={"ID":"59661663947a8edd85fd5fa4c740ff44","Type":"ContainerDied","Data":"d9afc4f07db171013d0ad9a79e71dd753566e5f6afc2ddc0397b92eaa5b1e275"} Nov 03 20:03:38.969992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.969993 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" event={"ID":"59661663947a8edd85fd5fa4c740ff44","Type":"ContainerStarted","Data":"fb89d284b9c2f57aef0ce2b778a77199c376e6284931637c15a649e66460a866"} Nov 03 20:03:38.970238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.970183 5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.991322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.991304 5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:03:38.991384 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.991338 5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:03:38.991934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:38.991922 5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:03:38.991968 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:38.991946 5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.077153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.077129 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.077240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.077158 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.077240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.077174 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/880c789d3d697e88c74ab8f8cf26d9d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" (UID: \"880c789d3d697e88c74ab8f8cf26d9d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.177693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.177672 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.177857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.177724 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.177857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.177747 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.177857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.177753 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59661663947a8edd85fd5fa4c740ff44-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" (UID: \"59661663947a8edd85fd5fa4c740ff44\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.177857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.177763 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/880c789d3d697e88c74ab8f8cf26d9d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" (UID: \"880c789d3d697e88c74ab8f8cf26d9d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.177857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.177814 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/880c789d3d697e88c74ab8f8cf26d9d3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-102.ec2.internal\" (UID: \"880c789d3d697e88c74ab8f8cf26d9d3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.616048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.616022 5043 apiserver.go:52] "Watching apiserver" Nov 03 20:03:39.667651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.667629 5043 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:03:39.668514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.668488 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td","openshift-multus/multus-additional-cni-plugins-d45vs","openshift-dns/node-resolver-zmtdq","openshift-ingress/router-default-7d68796f8f-nhd5b","openshift-ovn-kubernetes/ovnkube-node-2chqk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms","kube-system/kas-connection-checker-685895c7c9-tc29n","openshift-cluster-node-tuning-operator/tuned-qptp7","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7","kube-system/kube-apiserver-proxy-ip-10-0-140-102.ec2.internal","openshift-insights/insights-operator-7c4986d449-4md8j","openshift-insights/periodic-gathering-c6zf8-lzlj9","kube-system/global-pull-secret-syncer-kxtqd","openshift-dns/dns-default-5rssm","openshift-network-console/networking-console-plugin-86c478bb88-9mgnl","kube-system/konnectivity-agent-2bsd7","openshift-monitoring/cluster-monitoring-operator-884576c57-lv272","kube-system/kas-connection-checker-685895c7c9-htm8c","openshift-image-registry/image-registry-7c4d87c5c9-lhjzn","openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh","openshift-multus/network-metrics-daemon-8vwxs","openshift-network-diagnostics/network-check-source-6f78d59f99-59msh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt","openshift-console-operator/console-operator-7cd56b8dbc-6l254","openshift-network-operator/iptables-alerter-947nk","openshift-ingress-canary/ingress-canary-vl6h5","openshift-insights/insights-runtime-extractor-2nkf6","openshift-service-ca/service-ca-5fffc97bdf-xvg7r","kube-system/kas-connection-checker-685895c7c9-wh79h","openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt","openshift-network-diagnostics/network-check-target-7zg6w","openshift-image-registry/image-registry-6844b76f4-thtgj","openshift-multus/multus-dk9sk","openshift-image-registry/node-ca-bkb5w"] Nov 03 20:03:39.672160 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.672138 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.672289 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.672143 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.672595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.672572 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.674016 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.673998 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.676046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.676030 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.679368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.679352 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:39.681522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.681506 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.682684 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.682668 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.682836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.682810 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.682918 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.682879 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.683276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.683254 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.686108 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.686092 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.686648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.686630 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:03:39.686648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.686642 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:03:39.686648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.686638 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:03:39.691053 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.691031 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:03:39.692633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.692597 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:03:39.692728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.692670 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:03:39.694878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.693543 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8zgkz\"" Nov 03 20:03:39.694878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.693662 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:03:39.694878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.693762 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:03:39.694878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.693843 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:03:39.694878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.694745 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\"" Nov 03 20:03:39.695197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695171 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfgm6\"" Nov 03 20:03:39.695334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695320 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:03:39.695388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695340 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:03:39.695444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695387 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:03:39.695489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695466 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:03:39.695552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695535 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\"" Nov 03 20:03:39.695628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695608 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b4qhs\"" Nov 03 20:03:39.695677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695644 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hwv2m\"" Nov 03 20:03:39.695976 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.695943 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kzs97\"" Nov 03 20:03:39.696200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696184 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\"" Nov 03 20:03:39.696284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696205 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:03:39.696284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696267 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:03:39.696355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696251 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:03:39.696355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696284 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kzdq\"" Nov 03 20:03:39.696355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696304 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqzwq\"" Nov 03 20:03:39.696355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696310 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:03:39.696355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696314 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qw54\"" Nov 03 20:03:39.696565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696369 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4f2k\"" Nov 03 20:03:39.696565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696372 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-phn2h\"" Nov 03 20:03:39.696565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.696401 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:03:39.699472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.699456 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lf955\"" Nov 03 20:03:39.699472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.699461 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:03:39.704190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704175 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:03:39.704286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704256 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5kpvl\"" Nov 03 20:03:39.704344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704300 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.704344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704338 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:03:39.704447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704377 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.704447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704348 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:03:39.704540 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704526 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:03:39.704615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704591 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.704717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704701 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.704776 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704755 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.704776 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704758 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704797 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704815 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704823 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704815 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704816 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704837 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:03:39.704871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704869 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704872 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704889 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704911 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704924 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704924 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704940 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704953 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704970 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704971 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704987 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.704975 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705011 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705012 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705026 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705012 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705036 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705191 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705047 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705054 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705063 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705102 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705104 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705249 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705119 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705119 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705395 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705145 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v94zj\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705140 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705146 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705150 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705036 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705107 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705332 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705359 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705474 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705475 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705541 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705603 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705609 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705607 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-86559\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705619 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705777 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705830 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.705919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.705889 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t7mgh\"" Nov 03 20:03:39.708097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708066 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-95qsx\"" Nov 03 20:03:39.708287 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708259 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:03:39.708356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708318 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:03:39.708356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708345 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:03:39.708430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708345 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:03:39.708430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708378 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:03:39.708511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708448 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:03:39.708511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.708471 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cpwrs\"" Nov 03 20:03:39.733118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.733099 5043 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:03:39.735480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.735465 5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.756036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.756017 5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:03:39.756127 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.756060 5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-102.ec2.internal" Nov 03 20:03:39.780360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780339 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mng2w\" (UniqueName: \"kubernetes.io/projected/2bd39722-fd14-42d7-af96-2b37b2bba425-kube-api-access-mng2w\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.780429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780364 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8e9079-e494-4815-a2dc-fc3aa2763628-agent-certs\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:39.780429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780389 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-multus-certs\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.780429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780405 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.780565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780427 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bd39722-fd14-42d7-af96-2b37b2bba425-host-slash\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.780565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780533 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.780659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780589 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-bin\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.780659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780645 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.780744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780677 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-snapshots\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.780813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780749 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-snapshots\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.780900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780882 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.780931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780919 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjk6w\" (UniqueName: \"kubernetes.io/projected/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-api-access-pjk6w\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.780972 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780945 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-device-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.780972 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.780939 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8e9079-e494-4815-a2dc-fc3aa2763628-agent-certs\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:39.781060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781041 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.781120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781058 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.781120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781100 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-default-certificate\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.781184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781148 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70625344-a07f-4396-8843-f66a9f5421d1-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:39.781234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781215 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.781271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781234 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-service-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.781271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781262 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-host\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.781366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781280 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.781366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781300 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-etc-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.781366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781315 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70625344-a07f-4396-8843-f66a9f5421d1-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:39.781366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781339 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-ovn\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.781551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781369 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-systemd\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.781551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781389 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtdg\" (UniqueName: \"kubernetes.io/projected/70625344-a07f-4396-8843-f66a9f5421d1-kube-api-access-ddtdg\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:39.781551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781365 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-default-certificate\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.781551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781407 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.781551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781458 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtq4\" (UniqueName: \"kubernetes.io/projected/e341cc19-7823-4078-bd1c-1a7d3732882d-kube-api-access-9xtq4\") pod \"migrator-66df5d496b-qv7td\" (UID: \"e341cc19-7823-4078-bd1c-1a7d3732882d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781663 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781661 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781711 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysconfig\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781728 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-system-cni-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781736 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781746 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.781799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781783 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781807 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6z7\" (UniqueName: \"kubernetes.io/projected/aa34540b-36c1-4ef4-b4c9-e14a2419ee31-kube-api-access-7f6z7\") pod \"kas-connection-checker-685895c7c9-tc29n\" (UID: \"aa34540b-36c1-4ef4-b4c9-e14a2419ee31\") " pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781826 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-netns\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781846 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781869 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-systemd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781887 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-service-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781892 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vs5k\" (UniqueName: \"kubernetes.io/projected/331068c4-2a21-4ab6-9fbf-ea1ce605d767-kube-api-access-7vs5k\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781948 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-multus-daemon-config\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.781979 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782012 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782024 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782050 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzbb\" (UniqueName: \"kubernetes.io/projected/8ecf92d9-64ba-4348-9124-5f02d8032c9c-kube-api-access-9pzbb\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782102 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-sys-fs\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782129 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-config\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.782150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782153 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4ms\" (UniqueName: \"kubernetes.io/projected/fff6f118-0ff9-4478-b759-04e2d3f63420-kube-api-access-cv4ms\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782177 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782203 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqkl\" (UniqueName: \"kubernetes.io/projected/0138ff1f-c12a-446e-bd06-e62f8202a476-kube-api-access-4sqkl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782230 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-conf\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782234 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-multus-daemon-config\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782254 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-kubelet-config\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782403 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782411 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a28d511-b41b-4b0c-8819-6cba4a6620d8-config\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782452 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fff6f118-0ff9-4478-b759-04e2d3f63420-host\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782490 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782602 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782621 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-serving-cert\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782639 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782678 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t28n\" (UniqueName: \"kubernetes.io/projected/a22c4edd-da5c-44d4-a71c-6e162cf7f221-kube-api-access-8t28n\") pod \"volume-data-source-validator-58c9f5f46d-kvjms\" (UID: \"a22c4edd-da5c-44d4-a71c-6e162cf7f221\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782714 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-lib-modules\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.782784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782749 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-binary-copy\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782796 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-serving-cert\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782801 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782872 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782878 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782915 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-socket-dir-parent\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782946 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-kubelet\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782966 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-systemd-units\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.782981 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783018 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-binary-copy\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783035 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed430c0d-f5a9-4cf9-be18-182b648d8814-cert\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783057 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxnd\" (UniqueName: \"kubernetes.io/projected/e05b9d70-936b-44f5-b0f2-bc68492d7951-kube-api-access-zhxnd\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783086 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/967ed630-92ed-447b-b106-bb39959cbf0b-kube-api-access-v2l6w\") pod \"migrator-66df5d496b-msbdt\" (UID: \"967ed630-92ed-447b-b106-bb39959cbf0b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783128 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7jx\" (UniqueName: \"kubernetes.io/projected/1c49d972-a6cc-488a-90b7-8393e60a93a8-kube-api-access-8m7jx\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783154 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7bf\" (UniqueName: \"kubernetes.io/projected/31637ee3-ae67-4836-992c-3553f96a85e5-kube-api-access-tl7bf\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783178 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8e9079-e494-4815-a2dc-fc3aa2763628-konnectivity-ca\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783202 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db5689f3-b955-4d7c-b4b8-c203f7679fa1-serving-cert\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783226 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783252 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783268 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed430c0d-f5a9-4cf9-be18-182b648d8814-cert\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783280 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-tmp\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.783339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783302 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70625344-a07f-4396-8843-f66a9f5421d1-signing-key\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783364 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783367 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-tmp\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783403 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8e9079-e494-4815-a2dc-fc3aa2763628-konnectivity-ca\") pod \"konnectivity-agent-2bsd7\" (UID: \"0a8e9079-e494-4815-a2dc-fc3aa2763628\") " pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783484 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783501 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783556 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-k8s-cni-cncf-io\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783559 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70625344-a07f-4396-8843-f66a9f5421d1-signing-key\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783587 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-etc-kubernetes\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783601 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db5689f3-b955-4d7c-b4b8-c203f7679fa1-serving-cert\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783637 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0138ff1f-c12a-446e-bd06-e62f8202a476-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783655 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783682 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f94909-ffd1-4bea-ad03-10772d7775f3-data-volume\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783704 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783719 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-kubernetes\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783748 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783773 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783820 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mk2\" (UniqueName: \"kubernetes.io/projected/781d2937-5b44-4786-8c19-7629f4548399-kube-api-access-79mk2\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783835 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f94909-ffd1-4bea-ad03-10772d7775f3-data-volume\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783840 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783850 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-stats-auth\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.784174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.783877 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e05b9d70-936b-44f5-b0f2-bc68492d7951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784217 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784271 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-env-overrides\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784311 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-modprobe-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784348 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-run\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784380 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784414 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-cnibin\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784445 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-tmp\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784478 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff3150-0ab6-42d2-81ab-b03e7d944674-tmp-dir\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784484 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e05b9d70-936b-44f5-b0f2-bc68492d7951-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784270 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784589 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff3150-0ab6-42d2-81ab-b03e7d944674-tmp-dir\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784634 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442gc\" (UniqueName: \"kubernetes.io/projected/ed430c0d-f5a9-4cf9-be18-182b648d8814-kube-api-access-442gc\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784669 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f26e13-d1c9-4747-914f-af2fb15d7ed7-hosts-file\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784673 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-tmp\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784701 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-multus\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.784744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784738 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784767 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt24\" (UniqueName: \"kubernetes.io/projected/a7ff3150-0ab6-42d2-81ab-b03e7d944674-kube-api-access-9dt24\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784823 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784859 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-cni-binary-copy\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784897 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-script-lib\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784930 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25244768-eb42-47ea-a28d-20609fc12f42-original-pull-secret\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784937 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.784983 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h556\" (UniqueName: \"kubernetes.io/projected/2b318e41-0c85-4303-addc-336f59f4a4e7-kube-api-access-7h556\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785017 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a28d511-b41b-4b0c-8819-6cba4a6620d8-serving-cert\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785049 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fff6f118-0ff9-4478-b759-04e2d3f63420-serviceca\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785093 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-slash\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-var-lib-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.785182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785155 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovn-node-metrics-cert\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785186 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8g9\" (UniqueName: \"kubernetes.io/projected/26f26e13-d1c9-4747-914f-af2fb15d7ed7-kube-api-access-4p8g9\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785219 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-conf-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785232 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-env-overrides\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785245 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-node-log\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785276 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785339 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-sys\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.785504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785373 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0138ff1f-c12a-446e-bd06-e62f8202a476-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:39.785687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785565 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-stats-auth\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785374 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-registration-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785791 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovn-node-metrics-cert\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.785968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786139 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/781d2937-5b44-4786-8c19-7629f4548399-cni-binary-copy\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786259 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-script-lib\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786346 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db5689f3-b955-4d7c-b4b8-c203f7679fa1-config\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786565 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-system-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786617 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-cnibin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786667 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0138ff1f-c12a-446e-bd06-e62f8202a476-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786807 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fff6f118-0ff9-4478-b759-04e2d3f63420-serviceca\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786879 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tb5j\" (UniqueName: \"kubernetes.io/projected/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e-kube-api-access-5tb5j\") pod \"network-check-source-6f78d59f99-59msh\" (UID: \"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786921 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-dbus\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.786988 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787053 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787117 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18f94909-ffd1-4bea-ad03-10772d7775f3-crio-socket\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787159 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zw4h\" (UniqueName: \"kubernetes.io/projected/db5689f3-b955-4d7c-b4b8-c203f7679fa1-kube-api-access-9zw4h\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787198 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787207 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db5689f3-b955-4d7c-b4b8-c203f7679fa1-config\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787225 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0138ff1f-c12a-446e-bd06-e62f8202a476-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787505 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq5x\" (UniqueName: \"kubernetes.io/projected/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-kube-api-access-fkq5x\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787548 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787586 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6brv\" (UniqueName: \"kubernetes.io/projected/47ee139e-2eb1-4e93-9536-7c36c003ff40-kube-api-access-g6brv\") pod \"kas-connection-checker-685895c7c9-htm8c\" (UID: \"47ee139e-2eb1-4e93-9536-7c36c003ff40\") " pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787617 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-bin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787645 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-netns\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787758 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787790 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787824 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787856 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787886 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vrx\" (UniqueName: \"kubernetes.io/projected/dfd74d5b-c18f-42cb-be6d-416d2eb1c921-kube-api-access-k2vrx\") pod \"kas-connection-checker-685895c7c9-wh79h\" (UID: \"dfd74d5b-c18f-42cb-be6d-416d2eb1c921\") " pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787916 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-log-socket\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787949 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-var-lib-kubelet\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.787980 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt46\" (UniqueName: \"kubernetes.io/projected/edd592ba-dbd2-4026-906b-27478b901db5-kube-api-access-jrt46\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788013 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788043 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ff3150-0ab6-42d2-81ab-b03e7d944674-config-volume\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.788132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788093 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bd39722-fd14-42d7-af96-2b37b2bba425-iptables-alerter-script\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788256 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788298 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788330 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-config\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788361 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788390 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788432 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-kubelet\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788461 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-netd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788493 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-tuned\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788520 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-socket-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788552 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4p7d\" (UniqueName: \"kubernetes.io/projected/9a28d511-b41b-4b0c-8819-6cba4a6620d8-kube-api-access-h4p7d\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788584 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f26e13-d1c9-4747-914f-af2fb15d7ed7-tmp-dir\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788614 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-hostroot\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.788652 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq2q\" (UniqueName: \"kubernetes.io/projected/c2062bc3-f9db-48f9-aedb-b1367774ee26-kube-api-access-4nq2q\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.789246 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bd39722-fd14-42d7-af96-2b37b2bba425-iptables-alerter-script\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.789515 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.789880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.789883 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ff3150-0ab6-42d2-81ab-b03e7d944674-config-volume\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.790917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.790593 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.790917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.790726 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edd592ba-dbd2-4026-906b-27478b901db5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.790917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.790800 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-tuned\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.790917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.790903 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f26e13-d1c9-4747-914f-af2fb15d7ed7-tmp-dir\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.791221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.790777 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.791371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.791335 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a28d511-b41b-4b0c-8819-6cba4a6620d8-serving-cert\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.791878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.791853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/331068c4-2a21-4ab6-9fbf-ea1ce605d767-ovnkube-config\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.791951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.791927 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.800519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.800500 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjk6w\" (UniqueName: \"kubernetes.io/projected/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-api-access-pjk6w\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.800654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.800633 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mng2w\" (UniqueName: \"kubernetes.io/projected/2bd39722-fd14-42d7-af96-2b37b2bba425-kube-api-access-mng2w\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.801663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.801640 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtq4\" (UniqueName: \"kubernetes.io/projected/e341cc19-7823-4078-bd1c-1a7d3732882d-kube-api-access-9xtq4\") pod \"migrator-66df5d496b-qv7td\" (UID: \"e341cc19-7823-4078-bd1c-1a7d3732882d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qv7td" Nov 03 20:03:39.801849 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.801831 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") pod \"periodic-gathering-c6zf8-lzlj9\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:03:39.801904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.801835 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxnd\" (UniqueName: \"kubernetes.io/projected/e05b9d70-936b-44f5-b0f2-bc68492d7951-kube-api-access-zhxnd\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.801948 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.801938 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqkl\" (UniqueName: \"kubernetes.io/projected/0138ff1f-c12a-446e-bd06-e62f8202a476-kube-api-access-4sqkl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tchr7\" (UID: \"0138ff1f-c12a-446e-bd06-e62f8202a476\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" Nov 03 20:03:39.802186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.802170 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtdg\" (UniqueName: \"kubernetes.io/projected/70625344-a07f-4396-8843-f66a9f5421d1-kube-api-access-ddtdg\") pod \"service-ca-5fffc97bdf-xvg7r\" (UID: \"70625344-a07f-4396-8843-f66a9f5421d1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvg7r" Nov 03 20:03:39.804233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804113 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vs5k\" (UniqueName: \"kubernetes.io/projected/331068c4-2a21-4ab6-9fbf-ea1ce605d767-kube-api-access-7vs5k\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.804233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804137 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.804391 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804306 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.804464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804386 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t28n\" (UniqueName: \"kubernetes.io/projected/a22c4edd-da5c-44d4-a71c-6e162cf7f221-kube-api-access-8t28n\") pod \"volume-data-source-validator-58c9f5f46d-kvjms\" (UID: \"a22c4edd-da5c-44d4-a71c-6e162cf7f221\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kvjms" Nov 03 20:03:39.804598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804579 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/967ed630-92ed-447b-b106-bb39959cbf0b-kube-api-access-v2l6w\") pod \"migrator-66df5d496b-msbdt\" (UID: \"967ed630-92ed-447b-b106-bb39959cbf0b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-msbdt" Nov 03 20:03:39.804666 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804646 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4ms\" (UniqueName: \"kubernetes.io/projected/fff6f118-0ff9-4478-b759-04e2d3f63420-kube-api-access-cv4ms\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.804741 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804721 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzbb\" (UniqueName: \"kubernetes.io/projected/8ecf92d9-64ba-4348-9124-5f02d8032c9c-kube-api-access-9pzbb\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.804824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.804809 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mk2\" (UniqueName: \"kubernetes.io/projected/781d2937-5b44-4786-8c19-7629f4548399-kube-api-access-79mk2\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.805054 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.805032 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7jx\" (UniqueName: \"kubernetes.io/projected/1c49d972-a6cc-488a-90b7-8393e60a93a8-kube-api-access-8m7jx\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.805178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.805117 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7bf\" (UniqueName: \"kubernetes.io/projected/31637ee3-ae67-4836-992c-3553f96a85e5-kube-api-access-tl7bf\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.807041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.806959 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6z7\" (UniqueName: \"kubernetes.io/projected/aa34540b-36c1-4ef4-b4c9-e14a2419ee31-kube-api-access-7f6z7\") pod \"kas-connection-checker-685895c7c9-tc29n\" (UID: \"aa34540b-36c1-4ef4-b4c9-e14a2419ee31\") " pod="kube-system/kas-connection-checker-685895c7c9-tc29n" Nov 03 20:03:39.807139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807042 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h556\" (UniqueName: \"kubernetes.io/projected/2b318e41-0c85-4303-addc-336f59f4a4e7-kube-api-access-7h556\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:39.807321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807297 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-442gc\" (UniqueName: \"kubernetes.io/projected/ed430c0d-f5a9-4cf9-be18-182b648d8814-kube-api-access-442gc\") pod \"ingress-canary-vl6h5\" (UID: \"ed430c0d-f5a9-4cf9-be18-182b648d8814\") " pod="openshift-ingress-canary/ingress-canary-vl6h5" Nov 03 20:03:39.807393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807298 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpbh\" (UniqueName: \"kubernetes.io/projected/57170d5a-5e31-40d9-aebd-645e3c12c610-kube-api-access-vzpbh\") pod \"network-check-target-7zg6w\" (UID: \"57170d5a-5e31-40d9-aebd-645e3c12c610\") " pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:39.807577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807556 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zw4h\" (UniqueName: \"kubernetes.io/projected/db5689f3-b955-4d7c-b4b8-c203f7679fa1-kube-api-access-9zw4h\") pod \"service-ca-operator-544d749c8c-bdwvh\" (UID: \"db5689f3-b955-4d7c-b4b8-c203f7679fa1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" Nov 03 20:03:39.807648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807620 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt24\" (UniqueName: \"kubernetes.io/projected/a7ff3150-0ab6-42d2-81ab-b03e7d944674-kube-api-access-9dt24\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.807705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807624 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8g9\" (UniqueName: \"kubernetes.io/projected/26f26e13-d1c9-4747-914f-af2fb15d7ed7-kube-api-access-4p8g9\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.807799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.807778 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq5x\" (UniqueName: \"kubernetes.io/projected/ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5-kube-api-access-fkq5x\") pod \"insights-operator-7c4986d449-4md8j\" (UID: \"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5\") " pod="openshift-insights/insights-operator-7c4986d449-4md8j" Nov 03 20:03:39.808147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.808126 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.808198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.808173 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.808245 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.808225 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq2q\" (UniqueName: \"kubernetes.io/projected/c2062bc3-f9db-48f9-aedb-b1367774ee26-kube-api-access-4nq2q\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.808569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.808553 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt46\" (UniqueName: \"kubernetes.io/projected/edd592ba-dbd2-4026-906b-27478b901db5-kube-api-access-jrt46\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.808627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.808600 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6brv\" (UniqueName: \"kubernetes.io/projected/47ee139e-2eb1-4e93-9536-7c36c003ff40-kube-api-access-g6brv\") pod \"kas-connection-checker-685895c7c9-htm8c\" (UID: \"47ee139e-2eb1-4e93-9536-7c36c003ff40\") " pod="kube-system/kas-connection-checker-685895c7c9-htm8c" Nov 03 20:03:39.808850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.808834 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tb5j\" (UniqueName: \"kubernetes.io/projected/fa0036d1-fc35-4afe-9c9c-f252b3c34b3e-kube-api-access-5tb5j\") pod \"network-check-source-6f78d59f99-59msh\" (UID: \"fa0036d1-fc35-4afe-9c9c-f252b3c34b3e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-59msh" Nov 03 20:03:39.809492 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.809476 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vrx\" (UniqueName: \"kubernetes.io/projected/dfd74d5b-c18f-42cb-be6d-416d2eb1c921-kube-api-access-k2vrx\") pod \"kas-connection-checker-685895c7c9-wh79h\" (UID: \"dfd74d5b-c18f-42cb-be6d-416d2eb1c921\") " pod="kube-system/kas-connection-checker-685895c7c9-wh79h" Nov 03 20:03:39.809640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.809625 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4p7d\" (UniqueName: \"kubernetes.io/projected/9a28d511-b41b-4b0c-8819-6cba4a6620d8-kube-api-access-h4p7d\") pod \"console-operator-7cd56b8dbc-6l254\" (UID: \"9a28d511-b41b-4b0c-8819-6cba4a6620d8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:39.820569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.820526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:39.889683 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889635 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.889683 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889667 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-etc-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.889683 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889689 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-ovn\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889710 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-systemd\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889733 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysconfig\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889755 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-system-cni-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889766 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889779 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-netns\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889770 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-ovn\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889788 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-systemd\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889769 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-etc-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889804 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysconfig\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889817 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-systemd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889839 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-sys-fs\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889850 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-systemd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.889863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889856 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-conf\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889818 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-netns\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889804 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-system-cni-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889877 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-kubelet-config\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889913 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-sys-fs\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889918 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-kubelet-config\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889934 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fff6f118-0ff9-4478-b759-04e2d3f63420-host\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889957 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889981 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-lib-modules\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890005 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fff6f118-0ff9-4478-b759-04e2d3f63420-host\") pod \"node-ca-bkb5w\" (UID: \"fff6f118-0ff9-4478-b759-04e2d3f63420\") " pod="openshift-image-registry/node-ca-bkb5w" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890023 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890024 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.889997 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-conf\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890040 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890089 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-socket-dir-parent\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890113 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-kubelet\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.890120 5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890132 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-lib-modules\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890139 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-systemd-units\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.890155 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890163 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890182 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-socket-dir-parent\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890188 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-kubelet\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.890202 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:40.390181265 +0000 UTC m=+1.857136517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890210 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-run-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890196 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-systemd-units\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.890237 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:40.390230109 +0000 UTC m=+1.857185341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890220 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890259 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890273 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890296 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-k8s-cni-cncf-io\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890312 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-etc-kubernetes\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890332 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890352 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890375 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-etc-kubernetes\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890374 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-k8s-cni-cncf-io\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890377 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-kubernetes\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890421 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-kubernetes\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.890412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890424 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-modprobe-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890450 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-run\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890463 5043 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890473 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890503 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-modprobe-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890520 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-cnibin\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890529 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-run\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890492 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-etc-sysctl-d\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890569 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-cnibin\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.890572 5043 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890588 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f26e13-d1c9-4747-914f-af2fb15d7ed7-hosts-file\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.890626 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls podName:1c49d972-a6cc-488a-90b7-8393e60a93a8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:40.39061139 +0000 UTC m=+1.857566621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zqtvt" (UID: "1c49d972-a6cc-488a-90b7-8393e60a93a8") : secret "samples-operator-tls" not found Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890612 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-multus\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890662 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f26e13-d1c9-4747-914f-af2fb15d7ed7-hosts-file\") pod \"node-resolver-zmtdq\" (UID: \"26f26e13-d1c9-4747-914f-af2fb15d7ed7\") " pod="openshift-dns/node-resolver-zmtdq" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890662 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890698 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-slash\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890713 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-var-lib-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890665 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-multus\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890737 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-conf-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890763 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-node-log\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890772 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-var-lib-openvswitch\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890788 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890799 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-slash\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890809 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-sys\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890836 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-registration-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890810 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-multus-conf-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890847 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18f94909-ffd1-4bea-ad03-10772d7775f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890861 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-system-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890884 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-sys\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890886 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-cnibin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890906 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-registration-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890915 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-dbus\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890926 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-cnibin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890940 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890954 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-system-cni-dir\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890963 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18f94909-ffd1-4bea-ad03-10772d7775f3-crio-socket\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890840 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-node-log\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.890981 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25244768-eb42-47ea-a28d-20609fc12f42-dbus\") pod \"global-pull-secret-syncer-kxtqd\" (UID: \"25244768-eb42-47ea-a28d-20609fc12f42\") " pod="kube-system/global-pull-secret-syncer-kxtqd" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891000 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891010 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18f94909-ffd1-4bea-ad03-10772d7775f3-crio-socket\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891019 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edd592ba-dbd2-4026-906b-27478b901db5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d45vs\" (UID: \"edd592ba-dbd2-4026-906b-27478b901db5\") " pod="openshift-multus/multus-additional-cni-plugins-d45vs" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891032 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-bin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891050 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-netns\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891093 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891118 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891145 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-run-netns\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891158 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891184 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-log-socket\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891206 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-var-lib-kubelet\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.891231 5043 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891264 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-log-socket\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891117 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-var-lib-cni-bin\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:39.891351 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:40.391334596 +0000 UTC m=+1.858289834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : secret "metrics-daemon-secret" not found Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891236 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-kubelet\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891433 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-netd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891460 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-socket-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891532 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-kubelet\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891558 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-netd\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891595 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-var-lib-kubelet\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891655 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891683 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-socket-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891731 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-hostroot\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891778 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-multus-certs\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891805 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bd39722-fd14-42d7-af96-2b37b2bba425-host-slash\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891830 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2062bc3-f9db-48f9-aedb-b1367774ee26-service-ca-bundle\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-hostroot\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/781d2937-5b44-4786-8c19-7629f4548399-host-run-multus-certs\") pod \"multus-dk9sk\" (UID: \"781d2937-5b44-4786-8c19-7629f4548399\") " pod="openshift-multus/multus-dk9sk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891912 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bd39722-fd14-42d7-af96-2b37b2bba425-host-slash\") pod \"iptables-alerter-947nk\" (UID: \"2bd39722-fd14-42d7-af96-2b37b2bba425\") " pod="openshift-network-operator/iptables-alerter-947nk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.891971 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-bin\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892055 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-cni-bin\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892155 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-device-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892000 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31637ee3-ae67-4836-992c-3553f96a85e5-device-dir\") pod \"aws-ebs-csi-driver-node-rn5fk\" (UID: \"31637ee3-ae67-4836-992c-3553f96a85e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rn5fk" Nov 03 20:03:39.892185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892232 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892309 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/331068c4-2a21-4ab6-9fbf-ea1ce605d767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2chqk\" (UID: \"331068c4-2a21-4ab6-9fbf-ea1ce605d767\") " pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892386 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-host\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.892413 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ecf92d9-64ba-4348-9124-5f02d8032c9c-host\") pod \"tuned-qptp7\" (UID: \"8ecf92d9-64ba-4348-9124-5f02d8032c9c\") " pod="openshift-cluster-node-tuning-operator/tuned-qptp7" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.893498 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"image-registry-7c4d87c5c9-lhjzn\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.893690 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ff3150-0ab6-42d2-81ab-b03e7d944674-metrics-tls\") pod \"dns-default-5rssm\" (UID: \"a7ff3150-0ab6-42d2-81ab-b03e7d944674\") " pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.893976 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2062bc3-f9db-48f9-aedb-b1367774ee26-metrics-certs\") pod \"router-default-7d68796f8f-nhd5b\" (UID: \"c2062bc3-f9db-48f9-aedb-b1367774ee26\") " pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.894062 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85530c5c-c1ec-4d44-8a76-1ea11f34c7a2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9mgnl\" (UID: \"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.894797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.894150 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"image-registry-6844b76f4-thtgj\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.973438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.973421 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:39.974584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.974561 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:39.976035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.976018 5043 scope.go:122] "RemoveContainer" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" Nov 03 20:03:39.976309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.976291 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" Nov 03 20:03:39.981797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.981777 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:39.983158 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:39.983140 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:40.021934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.021655 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:40.122816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.122747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:40.221871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.221845 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:40.310093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.310057 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9mgnl"] Nov 03 20:03:40.313619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.313600 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d68796f8f-nhd5b"] Nov 03 20:03:40.316646 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:40.315444 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85530c5c_c1ec_4d44_8a76_1ea11f34c7a2.slice/crio-d32d60ac909bc53694405ebdb9f1ff40bc8e3fb986bca757599797b9249e3f3e WatchSource:0}: Error finding container d32d60ac909bc53694405ebdb9f1ff40bc8e3fb986bca757599797b9249e3f3e: Status 404 returned error can't find the container with id d32d60ac909bc53694405ebdb9f1ff40bc8e3fb986bca757599797b9249e3f3e Nov 03 20:03:40.318202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.318187 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:03:40.320044 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:40.320026 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2062bc3_f9db_48f9_aedb_b1367774ee26.slice/crio-b064d1199bb72146f8c3ab61708cb4f61c549ec2f7361a9e7869dc13d6674a2c WatchSource:0}: Error finding container b064d1199bb72146f8c3ab61708cb4f61c549ec2f7361a9e7869dc13d6674a2c: Status 404 returned error can't find the container with id b064d1199bb72146f8c3ab61708cb4f61c549ec2f7361a9e7869dc13d6674a2c Nov 03 20:03:40.337765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.337744 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rssm"] Nov 03 20:03:40.341299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.341280 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4d87c5c9-lhjzn"] Nov 03 20:03:40.349848 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:40.345947 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ff3150_0ab6_42d2_81ab_b03e7d944674.slice/crio-726baf5b7c832720ac83c43e6b6cda849ad2b6ae0f551aa10dbc80759f9614f3 WatchSource:0}: Error finding container 726baf5b7c832720ac83c43e6b6cda849ad2b6ae0f551aa10dbc80759f9614f3: Status 404 returned error can't find the container with id 726baf5b7c832720ac83c43e6b6cda849ad2b6ae0f551aa10dbc80759f9614f3 Nov 03 20:03:40.349848 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:40.346419 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259382b1_dd55_47ac_9744_c5b956a13ce7.slice/crio-571cdc48bd45a87bb87517c2e5fd4e525f62a3f2b18bd862f2cc2db32c6e7ebc WatchSource:0}: Error finding container 571cdc48bd45a87bb87517c2e5fd4e525f62a3f2b18bd862f2cc2db32c6e7ebc: Status 404 returned error can't find the container with id 571cdc48bd45a87bb87517c2e5fd4e525f62a3f2b18bd862f2cc2db32c6e7ebc Nov 03 20:03:40.352949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.352902 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6844b76f4-thtgj"] Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.401369 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.401396 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.401433 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.401490 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.401611 5043 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.401648 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:41.401635806 +0000 UTC m=+2.868591038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : secret "metrics-daemon-secret" not found Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.401687 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.401707 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:41.401700662 +0000 UTC m=+2.868655894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.401738 5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.401770 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:41.401750747 +0000 UTC m=+2.868705993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:40.404323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.404178 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c49d972-a6cc-488a-90b7-8393e60a93a8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zqtvt\" (UID: \"1c49d972-a6cc-488a-90b7-8393e60a93a8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:40.574633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.574594 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" Nov 03 20:03:40.625819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.625782 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:40.740566 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.740492 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rssm" event={"ID":"a7ff3150-0ab6-42d2-81ab-b03e7d944674","Type":"ContainerStarted","Data":"726baf5b7c832720ac83c43e6b6cda849ad2b6ae0f551aa10dbc80759f9614f3"} Nov 03 20:03:40.742058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.742012 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" event={"ID":"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2","Type":"ContainerStarted","Data":"d32d60ac909bc53694405ebdb9f1ff40bc8e3fb986bca757599797b9249e3f3e"} Nov 03 20:03:40.744089 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.744055 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:03:40.744720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.744582 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/1.log" Nov 03 20:03:40.744720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.744612 5043 generic.go:356] "Generic (PLEG): container finished" podID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" containerID="8fb2d4e25e70bcfbe3a5936681cd7bc5ced50ca4a1982c893c31b68c428710b5" exitCode=255 Nov 03 20:03:40.744720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.744692 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" event={"ID":"9a28d511-b41b-4b0c-8819-6cba4a6620d8","Type":"ContainerDied","Data":"8fb2d4e25e70bcfbe3a5936681cd7bc5ced50ca4a1982c893c31b68c428710b5"} Nov 03 20:03:40.744905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.744760 5043 scope.go:122] "RemoveContainer" containerID="da9d6d7e63a8a0528c7c35c3a762a426aff8af1f0d8cae7eb916987c4d3a7cd7" Nov 03 20:03:40.745109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.745090 5043 scope.go:122] "RemoveContainer" containerID="8fb2d4e25e70bcfbe3a5936681cd7bc5ced50ca4a1982c893c31b68c428710b5" Nov 03 20:03:40.745331 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:40.745308 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6l254_openshift-console-operator(9a28d511-b41b-4b0c-8819-6cba4a6620d8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" Nov 03 20:03:40.747236 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.747198 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" event={"ID":"c2062bc3-f9db-48f9-aedb-b1367774ee26","Type":"ContainerStarted","Data":"bd285e70ae5f48255f768d3526193aad0b3c77b4750091bbaaed0fcef44ae2d9"} Nov 03 20:03:40.747312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.747253 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" event={"ID":"c2062bc3-f9db-48f9-aedb-b1367774ee26","Type":"ContainerStarted","Data":"b064d1199bb72146f8c3ab61708cb4f61c549ec2f7361a9e7869dc13d6674a2c"} Nov 03 20:03:40.748822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.748798 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" event={"ID":"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8","Type":"ContainerStarted","Data":"12c93a548b3df4a3cd819d9dd2e635c1e7fac6a57b51b662d1d3440adf680056"} Nov 03 20:03:40.748896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.748831 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" event={"ID":"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8","Type":"ContainerStarted","Data":"9cc24a0257a31b855dd80f4094020e320d491b55191b4cb058a4223420272b4d"} Nov 03 20:03:40.750388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.750334 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" event={"ID":"259382b1-dd55-47ac-9744-c5b956a13ce7","Type":"ContainerStarted","Data":"0a578eddd2fabdb8c8a508b03676a43cd490e3c65b105813debc778e590e82a3"} Nov 03 20:03:40.750388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.750366 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" event={"ID":"259382b1-dd55-47ac-9744-c5b956a13ce7","Type":"ContainerStarted","Data":"571cdc48bd45a87bb87517c2e5fd4e525f62a3f2b18bd862f2cc2db32c6e7ebc"} Nov 03 20:03:40.877651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.877621 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt"] Nov 03 20:03:40.975000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.974977 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:40.977731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:40.977709 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:41.047470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.047380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:41.090202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.090172 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:03:41.218045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.217986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:41.412203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.412174 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:41.412324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.412210 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:41.412324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.412273 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:41.412449 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:41.412339 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:41.412449 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:41.412366 5043 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:03:41.412449 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:41.412374 5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:41.412449 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:41.412405 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.412386193 +0000 UTC m=+4.879341441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:41.412865 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:41.412845 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs podName:2b318e41-0c85-4303-addc-336f59f4a4e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.412412215 +0000 UTC m=+4.879367447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs") pod "network-metrics-daemon-8vwxs" (UID: "2b318e41-0c85-4303-addc-336f59f4a4e7") : secret "metrics-daemon-secret" not found Nov 03 20:03:41.412958 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:41.412880 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls podName:e05b9d70-936b-44f5-b0f2-bc68492d7951 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:43.412864803 +0000 UTC m=+4.879820040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lv272" (UID: "e05b9d70-936b-44f5-b0f2-bc68492d7951") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:03:41.757153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.756800 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9mgnl" event={"ID":"85530c5c-c1ec-4d44-8a76-1ea11f34c7a2","Type":"ContainerStarted","Data":"a83a5aa30df5345683464c4c13a11df7c50ce00070c4f3521fd9e0187fc0a28c"} Nov 03 20:03:41.758786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.758765 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:03:41.760329 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.760306 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:03:41.760716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.760696 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" event={"ID":"1c49d972-a6cc-488a-90b7-8393e60a93a8","Type":"ContainerStarted","Data":"7b843a54851f80de931f0daebe33096ba1c291603074c362a0ab14318c5d1dd9"} Nov 03 20:03:41.815087 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:41.815005 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:42.015220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.015138 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:42.232489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.232463 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:03:42.417965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.417929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:42.619807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.619780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:42.766410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.766377 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rssm" event={"ID":"a7ff3150-0ab6-42d2-81ab-b03e7d944674","Type":"ContainerStarted","Data":"bb00afc2996667d77206108dffc96ec54606a3314a996f18539cf2f07efb8e84"} Nov 03 20:03:42.766840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.766433 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rssm" event={"ID":"a7ff3150-0ab6-42d2-81ab-b03e7d944674","Type":"ContainerStarted","Data":"398f3cec96184ec87adb024e98706148ec6f9b9fc13c5c16dfdbd36dd0a7a557"} Nov 03 20:03:42.766840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.766478 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:42.768121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.768091 5043 generic.go:356] "Generic (PLEG): container finished" podID="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" containerID="65c7389b831e14b43dd73550000402ea93b50f6cec6d16fbf5ffc53264e40ef8" exitCode=0 Nov 03 20:03:42.768261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.768124 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4md8j" event={"ID":"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5","Type":"ContainerDied","Data":"65c7389b831e14b43dd73550000402ea93b50f6cec6d16fbf5ffc53264e40ef8"} Nov 03 20:03:42.768435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.768419 5043 scope.go:122] "RemoveContainer" containerID="65c7389b831e14b43dd73550000402ea93b50f6cec6d16fbf5ffc53264e40ef8" Nov 03 20:03:42.770131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.770102 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" event={"ID":"1c49d972-a6cc-488a-90b7-8393e60a93a8","Type":"ContainerStarted","Data":"c833547bc60b3958d069e2d21cd918bb1ae86dc2d14f5e9d6280957f28e2cf45"} Nov 03 20:03:42.770246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.770161 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zqtvt" event={"ID":"1c49d972-a6cc-488a-90b7-8393e60a93a8","Type":"ContainerStarted","Data":"26b66983adec0b39c087207851b57d3a5f788b29b05a6552b72204c347f50888"} Nov 03 20:03:42.770246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:42.770236 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:03:43.222410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.222369 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:43.304290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.304258 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:43.404487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.404462 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:43.430549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.430520 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:43.430698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.430681 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:43.430740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.430709 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:43.430839 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:43.430821 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:03:43.430926 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:43.430893 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls podName:18f94909-ffd1-4bea-ad03-10772d7775f3 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:47.430875145 +0000 UTC m=+8.897830393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nkf6" (UID: "18f94909-ffd1-4bea-ad03-10772d7775f3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:03:43.432922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.432893 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b318e41-0c85-4303-addc-336f59f4a4e7-metrics-certs\") pod \"network-metrics-daemon-8vwxs\" (UID: \"2b318e41-0c85-4303-addc-336f59f4a4e7\") " pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:43.433022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.433007 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e05b9d70-936b-44f5-b0f2-bc68492d7951-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lv272\" (UID: \"e05b9d70-936b-44f5-b0f2-bc68492d7951\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:43.575172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.575153 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" Nov 03 20:03:43.580182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.580155 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vwxs" Nov 03 20:03:43.776172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.776143 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4md8j" event={"ID":"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5","Type":"ContainerStarted","Data":"4fd5f8914247df1263b73a835228c5fdd7dd1d303ce0446de641a530d30e67d6"} Nov 03 20:03:43.777769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.777741 5043 generic.go:356] "Generic (PLEG): container finished" podID="0138ff1f-c12a-446e-bd06-e62f8202a476" containerID="184fa9e8dfa22b604fdb8860c7bd00a0dbd455df4359730ffae5e4f87b93a70b" exitCode=0 Nov 03 20:03:43.777888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.777811 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" event={"ID":"0138ff1f-c12a-446e-bd06-e62f8202a476","Type":"ContainerDied","Data":"184fa9e8dfa22b604fdb8860c7bd00a0dbd455df4359730ffae5e4f87b93a70b"} Nov 03 20:03:43.778286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.778269 5043 scope.go:122] "RemoveContainer" containerID="184fa9e8dfa22b604fdb8860c7bd00a0dbd455df4359730ffae5e4f87b93a70b" Nov 03 20:03:43.820051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.820021 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:43.873239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.873214 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lv272"] Nov 03 20:03:43.878163 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:43.878140 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05b9d70_936b_44f5_b0f2_bc68492d7951.slice/crio-4f62a42c6641543307328f67d4c34d9a1dca8fcc85803ea78b79756a68c2fd4c WatchSource:0}: Error finding container 4f62a42c6641543307328f67d4c34d9a1dca8fcc85803ea78b79756a68c2fd4c: Status 404 returned error can't find the container with id 4f62a42c6641543307328f67d4c34d9a1dca8fcc85803ea78b79756a68c2fd4c Nov 03 20:03:43.879479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:43.879456 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8vwxs"] Nov 03 20:03:43.885633 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:43.885608 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b318e41_0c85_4303_addc_336f59f4a4e7.slice/crio-d7470be9258e3ee98ff062a0ae7b66bf1ea2e2bc8f373f089114ea01934982fd WatchSource:0}: Error finding container d7470be9258e3ee98ff062a0ae7b66bf1ea2e2bc8f373f089114ea01934982fd: Status 404 returned error can't find the container with id d7470be9258e3ee98ff062a0ae7b66bf1ea2e2bc8f373f089114ea01934982fd Nov 03 20:03:44.034114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.034059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:44.165846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.165790 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:44.166168 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.166153 5043 scope.go:122] "RemoveContainer" containerID="8fb2d4e25e70bcfbe3a5936681cd7bc5ced50ca4a1982c893c31b68c428710b5" Nov 03 20:03:44.166389 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:44.166368 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6l254_openshift-console-operator(9a28d511-b41b-4b0c-8819-6cba4a6620d8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" Nov 03 20:03:44.418490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.418338 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:44.620130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.620034 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:44.785464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.785377 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" event={"ID":"e05b9d70-936b-44f5-b0f2-bc68492d7951","Type":"ContainerStarted","Data":"4f62a42c6641543307328f67d4c34d9a1dca8fcc85803ea78b79756a68c2fd4c"} Nov 03 20:03:44.786886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.786860 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vwxs" event={"ID":"2b318e41-0c85-4303-addc-336f59f4a4e7","Type":"ContainerStarted","Data":"d7470be9258e3ee98ff062a0ae7b66bf1ea2e2bc8f373f089114ea01934982fd"} Nov 03 20:03:44.789388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:44.789363 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tchr7" event={"ID":"0138ff1f-c12a-446e-bd06-e62f8202a476","Type":"ContainerStarted","Data":"80d13f743d68717e5c01b504b6ef613d3d0cba525feae0a1aa596b1e8940090e"} Nov 03 20:03:45.216817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.216785 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:45.337895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.337857 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:45.338111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.337984 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:03:45.339385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.339363 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d68796f8f-nhd5b" Nov 03 20:03:45.570929 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.570856 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:45.571367 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.571350 5043 scope.go:122] "RemoveContainer" containerID="8fb2d4e25e70bcfbe3a5936681cd7bc5ced50ca4a1982c893c31b68c428710b5" Nov 03 20:03:45.571600 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:03:45.571575 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6l254_openshift-console-operator(9a28d511-b41b-4b0c-8819-6cba4a6620d8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" Nov 03 20:03:45.795713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.795680 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lv272" event={"ID":"e05b9d70-936b-44f5-b0f2-bc68492d7951","Type":"ContainerStarted","Data":"11edc91cdbb099b2af6745c04d99a889729704b376bb87f00e9ee29859a6866f"} Nov 03 20:03:45.797637 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.797607 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vwxs" event={"ID":"2b318e41-0c85-4303-addc-336f59f4a4e7","Type":"ContainerStarted","Data":"05e7db91e9aa2a2f5d0599ee973c3a8016d89a8bc813759a2398fbb29f8a27d1"} Nov 03 20:03:45.797769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.797658 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vwxs" event={"ID":"2b318e41-0c85-4303-addc-336f59f4a4e7","Type":"ContainerStarted","Data":"177b0be70733f8e1ec947bdfcb599a8cb98f0a5bf340bf585c18fbe5db7b7f66"} Nov 03 20:03:45.820160 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:45.820134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:46.018198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:46.018162 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:46.167312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:46.167259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:46.270699 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:46.270628 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:46.421243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:46.421218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:46.535175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:46.535102 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:46.813616 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:46.813538 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:47.451700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:47.451673 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:03:47.466082 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:47.466048 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:47.468409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:47.468379 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18f94909-ffd1-4bea-ad03-10772d7775f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nkf6\" (UID: \"18f94909-ffd1-4bea-ad03-10772d7775f3\") " pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:47.483975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:47.483952 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nkf6" Nov 03 20:03:47.784317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:47.784227 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nkf6"] Nov 03 20:03:47.788515 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:47.788487 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f94909_ffd1_4bea_ad03_10772d7775f3.slice/crio-a206084fa45a698bf150b1da048ef41ef871a435c278bb623b5a2f51fa579d4c WatchSource:0}: Error finding container a206084fa45a698bf150b1da048ef41ef871a435c278bb623b5a2f51fa579d4c: Status 404 returned error can't find the container with id a206084fa45a698bf150b1da048ef41ef871a435c278bb623b5a2f51fa579d4c Nov 03 20:03:47.806419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:47.806393 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nkf6" event={"ID":"18f94909-ffd1-4bea-ad03-10772d7775f3","Type":"ContainerStarted","Data":"a206084fa45a698bf150b1da048ef41ef871a435c278bb623b5a2f51fa579d4c"} Nov 03 20:03:48.813013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:48.812937 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nkf6" event={"ID":"18f94909-ffd1-4bea-ad03-10772d7775f3","Type":"ContainerStarted","Data":"fa1f82fb66cfc32d43f376914ea2345fdad1be7eaf77838eea13a4afac804c04"} Nov 03 20:03:48.813013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:48.812990 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nkf6" event={"ID":"18f94909-ffd1-4bea-ad03-10772d7775f3","Type":"ContainerStarted","Data":"c23611afddeb79200521410346fe2e5a2d458144c5b3e006bf658d2640d51ce8"} Nov 03 20:03:49.556605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:49.556576 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:49.556832 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:49.556817 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:03:49.819104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:49.819023 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nkf6" event={"ID":"18f94909-ffd1-4bea-ad03-10772d7775f3","Type":"ContainerStarted","Data":"090c9d598b4f0d5f968ce967070ec943ec456fbae4de239402fc3c3157edece7"} Nov 03 20:03:50.670145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:50.670116 5043 scope.go:122] "RemoveContainer" containerID="8fb2d4e25e70bcfbe3a5936681cd7bc5ced50ca4a1982c893c31b68c428710b5" Nov 03 20:03:50.826592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:50.826562 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:03:50.827060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:50.826690 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" event={"ID":"9a28d511-b41b-4b0c-8819-6cba4a6620d8","Type":"ContainerStarted","Data":"ec2c771c17e9c30147b836beb63039648d941a3633050eed38c7cf461e65e6a4"} Nov 03 20:03:50.827170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:50.827123 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:50.828097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:50.828049 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-6l254 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Nov 03 20:03:50.828201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:50.828108 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" podUID="9a28d511-b41b-4b0c-8819-6cba4a6620d8" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Nov 03 20:03:51.420674 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.420641 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:51.522508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.522456 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:51.618876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.618844 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:51.718767 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.718681 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:51.819965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.819935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:51.834959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.834940 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6l254" Nov 03 20:03:51.920784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:51.920756 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:52.135918 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.135894 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2chqk" Nov 03 20:03:52.443281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.443206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:52.627346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.627316 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:52.683435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.683390 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6844b76f4-thtgj"] Nov 03 20:03:52.695965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.695892 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5phwq"] Nov 03 20:03:52.699171 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.699146 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:52.702163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.702111 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62"] Nov 03 20:03:52.704440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.704406 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:52.719980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.719960 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-2cl9k\"" Nov 03 20:03:52.720153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.719960 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:03:52.720153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.720113 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:03:52.725998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.725968 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5phwq"] Nov 03 20:03:52.732632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.732609 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62"] Nov 03 20:03:52.780560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.780532 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5rssm" Nov 03 20:03:52.782670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.782644 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dc8df666f-5wrpc"] Nov 03 20:03:52.785642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.785620 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.790794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.790768 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:52.810360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.810328 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc8df666f-5wrpc"] Nov 03 20:03:52.811671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.811648 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1161d01c-a20b-4036-be74-22978c149f7d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-x2m62\" (UID: \"1161d01c-a20b-4036-be74-22978c149f7d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:52.811775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.811698 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qlz\" (UniqueName: \"kubernetes.io/projected/2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef-kube-api-access-75qlz\") pod \"downloads-6dd789c8b7-5phwq\" (UID: \"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef\") " pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:52.811775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.811732 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef-tmp\") pod \"downloads-6dd789c8b7-5phwq\" (UID: \"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef\") " pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:52.819552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.819512 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:03:52.819697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.819663 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kpvn4\"" Nov 03 20:03:52.912603 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912556 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1161d01c-a20b-4036-be74-22978c149f7d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-x2m62\" (UID: \"1161d01c-a20b-4036-be74-22978c149f7d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912652 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qlz\" (UniqueName: \"kubernetes.io/projected/2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef-kube-api-access-75qlz\") pod \"downloads-6dd789c8b7-5phwq\" (UID: \"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef\") " pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912692 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-registry-tls\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912721 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfx6x\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-kube-api-access-rfx6x\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912759 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef-tmp\") pod \"downloads-6dd789c8b7-5phwq\" (UID: \"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef\") " pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912793 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-bound-sa-token\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.912877 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c7fb94d-1281-4d31-8413-b97a6c0389f8-image-registry-private-configuration\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.913027 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c7fb94d-1281-4d31-8413-b97a6c0389f8-trusted-ca\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.913126 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c7fb94d-1281-4d31-8413-b97a6c0389f8-installation-pull-secrets\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.913178 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c7fb94d-1281-4d31-8413-b97a6c0389f8-registry-certificates\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.913201 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c7fb94d-1281-4d31-8413-b97a6c0389f8-ca-trust-extracted\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:52.913544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.913340 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef-tmp\") pod \"downloads-6dd789c8b7-5phwq\" (UID: \"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef\") " pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:52.914612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.914517 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:52.915575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.915550 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1161d01c-a20b-4036-be74-22978c149f7d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-x2m62\" (UID: \"1161d01c-a20b-4036-be74-22978c149f7d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:52.932277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:52.932253 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qlz\" (UniqueName: \"kubernetes.io/projected/2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef-kube-api-access-75qlz\") pod \"downloads-6dd789c8b7-5phwq\" (UID: \"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef\") " pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:53.013675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013598 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-bound-sa-token\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013656 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c7fb94d-1281-4d31-8413-b97a6c0389f8-image-registry-private-configuration\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013679 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c7fb94d-1281-4d31-8413-b97a6c0389f8-trusted-ca\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013698 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c7fb94d-1281-4d31-8413-b97a6c0389f8-installation-pull-secrets\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013716 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c7fb94d-1281-4d31-8413-b97a6c0389f8-registry-certificates\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013733 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c7fb94d-1281-4d31-8413-b97a6c0389f8-ca-trust-extracted\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013815 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-registry-tls\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.013874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.013841 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfx6x\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-kube-api-access-rfx6x\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.014447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.014381 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c7fb94d-1281-4d31-8413-b97a6c0389f8-ca-trust-extracted\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.014671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.014649 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c7fb94d-1281-4d31-8413-b97a6c0389f8-registry-certificates\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.015455 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.015429 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c7fb94d-1281-4d31-8413-b97a6c0389f8-trusted-ca\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.015593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.015578 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:03:53.016309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.016277 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c7fb94d-1281-4d31-8413-b97a6c0389f8-installation-pull-secrets\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.016407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.016306 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c7fb94d-1281-4d31-8413-b97a6c0389f8-image-registry-private-configuration\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.016718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.016698 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-registry-tls\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.023104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.023084 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:53.039673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.039621 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfx6x\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-kube-api-access-rfx6x\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.061898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.061872 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c7fb94d-1281-4d31-8413-b97a6c0389f8-bound-sa-token\") pod \"image-registry-dc8df666f-5wrpc\" (UID: \"6c7fb94d-1281-4d31-8413-b97a6c0389f8\") " pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.095483 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.086795 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:53.101244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.101213 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.338008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.337977 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62"] Nov 03 20:03:53.343331 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:53.343303 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1161d01c_a20b_4036_be74_22978c149f7d.slice/crio-b4d8157fde3a7a384e68e4a4584f6a9d28785f7f7ebe4c0a6c79f8063dea35c7 WatchSource:0}: Error finding container b4d8157fde3a7a384e68e4a4584f6a9d28785f7f7ebe4c0a6c79f8063dea35c7: Status 404 returned error can't find the container with id b4d8157fde3a7a384e68e4a4584f6a9d28785f7f7ebe4c0a6c79f8063dea35c7 Nov 03 20:03:53.351920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.351894 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5phwq"] Nov 03 20:03:53.357758 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:53.357729 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae1a8b7_4dd4_4e7a_8f29_c08ec80f10ef.slice/crio-5101a12f35daf76d16583c926b753cd228fe0dccbe54ba06aca5f2ebbeadb55e WatchSource:0}: Error finding container 5101a12f35daf76d16583c926b753cd228fe0dccbe54ba06aca5f2ebbeadb55e: Status 404 returned error can't find the container with id 5101a12f35daf76d16583c926b753cd228fe0dccbe54ba06aca5f2ebbeadb55e Nov 03 20:03:53.422860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.422821 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc8df666f-5wrpc"] Nov 03 20:03:53.427128 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:03:53.427097 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7fb94d_1281_4d31_8413_b97a6c0389f8.slice/crio-2f78ce1783284366b5e0342b0f4daadafeb835b93a0a6cf495de9e651e14bb29 WatchSource:0}: Error finding container 2f78ce1783284366b5e0342b0f4daadafeb835b93a0a6cf495de9e651e14bb29: Status 404 returned error can't find the container with id 2f78ce1783284366b5e0342b0f4daadafeb835b93a0a6cf495de9e651e14bb29 Nov 03 20:03:53.427574 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.427524 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:53.527565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.527532 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:53.856784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.856746 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" event={"ID":"1161d01c-a20b-4036-be74-22978c149f7d","Type":"ContainerStarted","Data":"b4d8157fde3a7a384e68e4a4584f6a9d28785f7f7ebe4c0a6c79f8063dea35c7"} Nov 03 20:03:53.858467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.858435 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5phwq" event={"ID":"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef","Type":"ContainerStarted","Data":"5101a12f35daf76d16583c926b753cd228fe0dccbe54ba06aca5f2ebbeadb55e"} Nov 03 20:03:53.860320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.860284 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" event={"ID":"6c7fb94d-1281-4d31-8413-b97a6c0389f8","Type":"ContainerStarted","Data":"3867ea3b3bf12b4d83258bc7de0c1f95b36642c858d67cac62ffcfff92b86854"} Nov 03 20:03:53.860444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.860348 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" event={"ID":"6c7fb94d-1281-4d31-8413-b97a6c0389f8","Type":"ContainerStarted","Data":"2f78ce1783284366b5e0342b0f4daadafeb835b93a0a6cf495de9e651e14bb29"} Nov 03 20:03:53.860527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.860501 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:03:53.908098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:53.902998 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" podStartSLOduration=1.9029857369999998 podStartE2EDuration="1.902985737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.902713923 +0000 UTC m=+15.369669178" watchObservedRunningTime="2026-11-03 20:03:53.902985737 +0000 UTC m=+15.369940968" Nov 03 20:03:54.871252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:54.871219 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" event={"ID":"1161d01c-a20b-4036-be74-22978c149f7d","Type":"ContainerStarted","Data":"3a28ba35bbda6262dd1bb9f40844022955f70c79d9de6dce3fc5acdcc774b41c"} Nov 03 20:03:54.871703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:54.871505 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:54.876787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:54.876761 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" Nov 03 20:03:54.908638 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:54.908592 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-x2m62" podStartSLOduration=1.540765293 podStartE2EDuration="2.908580371s" totalImagesPullingTime="1.367815078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:54.907328753 +0000 UTC m=+16.374284008" watchObservedRunningTime="2026-11-03 20:03:54.908580371 +0000 UTC m=+16.375535605" Nov 03 20:03:54.986366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:54.986272 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:55.620639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:55.620595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:55.723982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:55.723943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:55.865982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:55.865939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:56.254647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:56.254607 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:58.607545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:58.607510 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:58.767615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:58.767581 5043 patch_prober.go:29] interesting pod/image-registry-7c4d87c5c9-lhjzn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:03:58.767790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:58.767632 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" podUID="259382b1-dd55-47ac-9744-c5b956a13ce7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:03:59.381334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:03:59.381299 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2bsd7" Nov 03 20:04:00.027275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.027234 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.127989 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.127933 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.236524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.236490 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.332818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.332785 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-87rfg"] Nov 03 20:04:00.338002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.337683 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.345281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.345181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.398557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.398518 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:04:00.398680 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.398559 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:04:00.398680 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.398656 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89ghp\"" Nov 03 20:04:00.398784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.398770 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:04:00.398828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.398817 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:04:00.443778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.443750 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.492933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.492899 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee70c47d-21c7-433d-9d08-87df444b9235-metrics-client-ca\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493087 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.492997 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-accelerators-collector-config\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493096 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbf4\" (UniqueName: \"kubernetes.io/projected/ee70c47d-21c7-433d-9d08-87df444b9235-kube-api-access-jtbf4\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493135 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-wtmp\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493161 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493302 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493257 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-textfile\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493302 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493292 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-root\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493390 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493318 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-sys\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.493434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.493385 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-tls\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.546838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.546812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.594303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594240 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee70c47d-21c7-433d-9d08-87df444b9235-metrics-client-ca\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594280 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-accelerators-collector-config\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594354 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbf4\" (UniqueName: \"kubernetes.io/projected/ee70c47d-21c7-433d-9d08-87df444b9235-kube-api-access-jtbf4\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594410 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-wtmp\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594439 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594497 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-textfile\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594527 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-root\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594551 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-sys\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594573 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-tls\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.594931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.594907 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-accelerators-collector-config\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.595104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.595058 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-wtmp\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.595183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.595121 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-root\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.595262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.595232 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee70c47d-21c7-433d-9d08-87df444b9235-sys\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.595428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.595403 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-textfile\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.595671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.595651 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee70c47d-21c7-433d-9d08-87df444b9235-metrics-client-ca\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.600563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.599584 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-tls\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.609059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.606391 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee70c47d-21c7-433d-9d08-87df444b9235-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.613285 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.613260 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbf4\" (UniqueName: \"kubernetes.io/projected/ee70c47d-21c7-433d-9d08-87df444b9235-kube-api-access-jtbf4\") pod \"node-exporter-87rfg\" (UID: \"ee70c47d-21c7-433d-9d08-87df444b9235\") " pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.650353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.650313 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.655044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.655018 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-87rfg" Nov 03 20:04:00.686195 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:04:00.686163 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee70c47d_21c7_433d_9d08_87df444b9235.slice/crio-fd16e7139d1de5d18ec4ccbd32f4eb479ea35c2d464d9a39661eed33ec66b900 WatchSource:0}: Error finding container fd16e7139d1de5d18ec4ccbd32f4eb479ea35c2d464d9a39661eed33ec66b900: Status 404 returned error can't find the container with id fd16e7139d1de5d18ec4ccbd32f4eb479ea35c2d464d9a39661eed33ec66b900 Nov 03 20:04:00.747163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.747137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.900594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:00.900522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-87rfg" event={"ID":"ee70c47d-21c7-433d-9d08-87df444b9235","Type":"ContainerStarted","Data":"fd16e7139d1de5d18ec4ccbd32f4eb479ea35c2d464d9a39661eed33ec66b900"} Nov 03 20:04:01.404144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:01.404029 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:01.585551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:01.585385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:01.685595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:01.685503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:02.690347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:02.690318 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:04:02.775047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:02.775017 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:04:02.911912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:02.911865 5043 generic.go:356] "Generic (PLEG): container finished" podID="ee70c47d-21c7-433d-9d08-87df444b9235" containerID="f3ee02d68210a3e90eaf0b753c66cd7293492e00e320d5438a3b606ee0b57a7e" exitCode=0 Nov 03 20:04:02.912121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:02.911933 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-87rfg" event={"ID":"ee70c47d-21c7-433d-9d08-87df444b9235","Type":"ContainerDied","Data":"f3ee02d68210a3e90eaf0b753c66cd7293492e00e320d5438a3b606ee0b57a7e"} Nov 03 20:04:03.126112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.126057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:03.221302 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.221219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:03.525663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.525575 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:03.626692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.626645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:03.726488 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.726450 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:03.919627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.919588 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-87rfg" event={"ID":"ee70c47d-21c7-433d-9d08-87df444b9235","Type":"ContainerStarted","Data":"db0e870e3627910430ed22833389ca01281e5c64f13054a0aeb1a5f95986ff8b"} Nov 03 20:04:03.919815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.919680 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-87rfg" event={"ID":"ee70c47d-21c7-433d-9d08-87df444b9235","Type":"ContainerStarted","Data":"b304cab3e3e754d1efc5951506b1c68b11b245528ccde91edd9943e211f995e9"} Nov 03 20:04:03.990180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:03.990132 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-87rfg" podStartSLOduration=2.548760291 podStartE2EDuration="3.990095485s" totalImagesPullingTime="1.350774245s" totalInitContainerRuntime="90.560949ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:03.989432808 +0000 UTC m=+25.456388061" watchObservedRunningTime="2026-11-03 20:04:03.990095485 +0000 UTC m=+25.457050738" Nov 03 20:04:05.104445 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:05.104392 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:05.200615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:05.200573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:05.291233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:05.291194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:05.630818 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:04:05.630786 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:04:06.142328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:06.142290 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:06.242005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:06.241968 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:07.174714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:07.174675 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:07.444130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:07.444048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:07.454452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:07.454429 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7zg6w" Nov 03 20:04:07.544541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:07.544506 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:08.421202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:08.421065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:08.504971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:08.504936 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:10.960749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:10.960710 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5phwq" event={"ID":"2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef","Type":"ContainerStarted","Data":"6b6d08160b8d8e10f4bd0dd1118c2d22be2dd6518038682c4f05c81cb73b2bab"} Nov 03 20:04:10.961256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:10.960954 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:04:10.976099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:10.976044 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5phwq" Nov 03 20:04:11.024915 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:11.024864 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-5phwq" podStartSLOduration=2.226557502 podStartE2EDuration="19.0248491s" totalImagesPullingTime="16.798291598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:11.011692085 +0000 UTC m=+32.478647360" watchObservedRunningTime="2026-11-03 20:04:11.0248491 +0000 UTC m=+32.491804354" Nov 03 20:04:13.039876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:13.039838 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:13.149797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:13.149762 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:14.876718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:14.876688 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dc8df666f-5wrpc" Nov 03 20:04:14.954306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:14.954271 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4d87c5c9-lhjzn"] Nov 03 20:04:15.044707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:15.044662 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:15.144859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:15.144777 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:15.246109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:15.246043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:15.348061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:15.348017 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:16.460785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:16.460744 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:16.561819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:16.561773 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:16.663509 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:16.663474 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:17.710017 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:17.709976 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" podUID="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" containerName="registry" containerID="cri-o://12c93a548b3df4a3cd819d9dd2e635c1e7fac6a57b51b662d1d3440adf680056" gracePeriod=30 Nov 03 20:04:17.997190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:17.997162 5043 generic.go:356] "Generic (PLEG): container finished" podID="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" containerID="12c93a548b3df4a3cd819d9dd2e635c1e7fac6a57b51b662d1d3440adf680056" exitCode=0 Nov 03 20:04:17.997315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:17.997215 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" event={"ID":"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8","Type":"ContainerDied","Data":"12c93a548b3df4a3cd819d9dd2e635c1e7fac6a57b51b662d1d3440adf680056"} Nov 03 20:04:17.997315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:17.997259 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" event={"ID":"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8","Type":"ContainerDied","Data":"9cc24a0257a31b855dd80f4094020e320d491b55191b4cb058a4223420272b4d"} Nov 03 20:04:17.997315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:17.997279 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cc24a0257a31b855dd80f4094020e320d491b55191b4cb058a4223420272b4d" Nov 03 20:04:18.028247 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.028218 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:04:18.149951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.149918 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.149978 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.150123 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.150193 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.150254 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.150284 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.150331 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.150400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.150374 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") pod \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\" (UID: \"eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8\") " Nov 03 20:04:18.152038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.152003 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:18.153090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.153031 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:18.153386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.153348 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:18.153502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.153474 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:18.154741 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.154702 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:18.155001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.154977 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "kube-api-access-xksm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:18.156568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.156543 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:18.159203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.159178 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted" pod "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" (UID: "eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:18.251249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251186 5043 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-installation-pull-secrets\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251221 5043 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-ca-trust-extracted\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251239 5043 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-image-registry-private-configuration\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251253 5043 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-certificates\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251263 5043 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-bound-sa-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251272 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xksm5\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-kube-api-access-xksm5\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251280 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-trusted-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:18.251453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:18.251290 5043 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8-registry-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:19.002105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:19.002020 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6844b76f4-thtgj" Nov 03 20:04:19.062304 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:19.062272 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6844b76f4-thtgj"] Nov 03 20:04:19.073985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:19.073955 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6844b76f4-thtgj"] Nov 03 20:04:19.623157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:19.623119 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:19.723093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:19.723034 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:20.675672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:20.675638 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" path="/var/lib/kubelet/pods/eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8/volumes" Nov 03 20:04:22.483710 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:22.483677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:22.590150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:22.590119 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:24.635579 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:24.635547 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:24.720161 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:24.720135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:28.238034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:28.237997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:28.352686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:28.352653 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:29.428764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:29.428732 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:29.511839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:29.511808 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:33.472736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:33.472689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:33.554369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:33.554336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:35.626100 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:04:35.626043 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:04:36.166444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:36.166409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:37.439253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:37.439219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:37.540852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:37.540818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:39.979981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:39.979917 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" podUID="259382b1-dd55-47ac-9744-c5b956a13ce7" containerName="registry" containerID="cri-o://0a578eddd2fabdb8c8a508b03676a43cd490e3c65b105813debc778e590e82a3" gracePeriod=30 Nov 03 20:04:40.094874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.094843 5043 generic.go:356] "Generic (PLEG): container finished" podID="259382b1-dd55-47ac-9744-c5b956a13ce7" containerID="0a578eddd2fabdb8c8a508b03676a43cd490e3c65b105813debc778e590e82a3" exitCode=0 Nov 03 20:04:40.094982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.094912 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" event={"ID":"259382b1-dd55-47ac-9744-c5b956a13ce7","Type":"ContainerDied","Data":"0a578eddd2fabdb8c8a508b03676a43cd490e3c65b105813debc778e590e82a3"} Nov 03 20:04:40.244022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.243965 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:04:40.320812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.320784 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.320928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.320858 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.320981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.320961 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.321065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321026 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.321065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321058 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.321180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321103 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.321180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321161 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.321283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321206 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") pod \"259382b1-dd55-47ac-9744-c5b956a13ce7\" (UID: \"259382b1-dd55-47ac-9744-c5b956a13ce7\") " Nov 03 20:04:40.321470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321403 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:40.321600 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.321475 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:40.323626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.323604 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "kube-api-access-p9sbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:40.323814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.323788 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:40.323950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.323930 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:40.323950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.323940 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:40.324043 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.323949 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:40.330420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.330396 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted" pod "259382b1-dd55-47ac-9744-c5b956a13ce7" (UID: "259382b1-dd55-47ac-9744-c5b956a13ce7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:40.422424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422402 5043 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-certificates\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422422 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/259382b1-dd55-47ac-9744-c5b956a13ce7-trusted-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422432 5043 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-registry-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422440 5043 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-image-registry-private-configuration\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422450 5043 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/259382b1-dd55-47ac-9744-c5b956a13ce7-installation-pull-secrets\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422458 5043 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-bound-sa-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422466 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9sbx\" (UniqueName: \"kubernetes.io/projected/259382b1-dd55-47ac-9744-c5b956a13ce7-kube-api-access-p9sbx\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:40.422558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:40.422474 5043 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/259382b1-dd55-47ac-9744-c5b956a13ce7-ca-trust-extracted\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:41.100140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:41.100106 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" event={"ID":"259382b1-dd55-47ac-9744-c5b956a13ce7","Type":"ContainerDied","Data":"571cdc48bd45a87bb87517c2e5fd4e525f62a3f2b18bd862f2cc2db32c6e7ebc"} Nov 03 20:04:41.100568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:41.100168 5043 scope.go:122] "RemoveContainer" containerID="0a578eddd2fabdb8c8a508b03676a43cd490e3c65b105813debc778e590e82a3" Nov 03 20:04:41.100568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:41.100116 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4d87c5c9-lhjzn" Nov 03 20:04:41.163799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:41.163764 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4d87c5c9-lhjzn"] Nov 03 20:04:41.174869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:41.174846 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c4d87c5c9-lhjzn"] Nov 03 20:04:42.675153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:42.675120 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259382b1-dd55-47ac-9744-c5b956a13ce7" path="/var/lib/kubelet/pods/259382b1-dd55-47ac-9744-c5b956a13ce7/volumes" Nov 03 20:04:44.117879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:44.117849 5043 generic.go:356] "Generic (PLEG): container finished" podID="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" containerID="4fd5f8914247df1263b73a835228c5fdd7dd1d303ce0446de641a530d30e67d6" exitCode=0 Nov 03 20:04:44.118287 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:44.117923 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4md8j" event={"ID":"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5","Type":"ContainerDied","Data":"4fd5f8914247df1263b73a835228c5fdd7dd1d303ce0446de641a530d30e67d6"} Nov 03 20:04:44.118287 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:44.117980 5043 scope.go:122] "RemoveContainer" containerID="65c7389b831e14b43dd73550000402ea93b50f6cec6d16fbf5ffc53264e40ef8" Nov 03 20:04:44.118394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:44.118311 5043 scope.go:122] "RemoveContainer" containerID="4fd5f8914247df1263b73a835228c5fdd7dd1d303ce0446de641a530d30e67d6" Nov 03 20:04:44.118519 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:04:44.118498 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-4md8j_openshift-insights(ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5)\"" pod="openshift-insights/insights-operator-7c4986d449-4md8j" podUID="ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5" Nov 03 20:04:45.519406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:45.519368 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:45.859282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:45.859246 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:46.447537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:46.447503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:46.547466 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:46.547437 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:46.647264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:46.647238 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:54.469840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:54.469799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:54.776244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:54.776172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:57.178226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:57.178199 5043 generic.go:356] "Generic (PLEG): container finished" podID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" containerID="fe3adbd709414b43e3046dfce686930becb6b485cafc2ac8e0a451c049582c99" exitCode=0 Nov 03 20:04:57.178537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:57.178268 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" event={"ID":"aa7644a3-9a1a-47a8-82da-44de32fa7ee2","Type":"ContainerDied","Data":"fe3adbd709414b43e3046dfce686930becb6b485cafc2ac8e0a451c049582c99"} Nov 03 20:04:58.321061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.321039 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:04:58.441908 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.441850 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") pod \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " Nov 03 20:04:58.441908 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.441897 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") pod \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " Nov 03 20:04:58.442060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.441932 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") pod \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " Nov 03 20:04:58.442060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.441955 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") pod \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\" (UID: \"aa7644a3-9a1a-47a8-82da-44de32fa7ee2\") " Nov 03 20:04:58.442352 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.442317 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path" pod "aa7644a3-9a1a-47a8-82da-44de32fa7ee2" (UID: "aa7644a3-9a1a-47a8-82da-44de32fa7ee2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:58.442467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.442356 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle" pod "aa7644a3-9a1a-47a8-82da-44de32fa7ee2" (UID: "aa7644a3-9a1a-47a8-82da-44de32fa7ee2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:58.442467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.442366 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle" pod "aa7644a3-9a1a-47a8-82da-44de32fa7ee2" (UID: "aa7644a3-9a1a-47a8-82da-44de32fa7ee2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:58.444121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.444098 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq" pod "aa7644a3-9a1a-47a8-82da-44de32fa7ee2" (UID: "aa7644a3-9a1a-47a8-82da-44de32fa7ee2"). InnerVolumeSpecName "kube-api-access-wm5gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:58.543026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.543003 5043 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-service-ca-bundle\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:58.543026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.543024 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5gq\" (UniqueName: \"kubernetes.io/projected/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-kube-api-access-wm5gq\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:58.543167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.543034 5043 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-archives-path\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:58.543167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.543043 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa7644a3-9a1a-47a8-82da-44de32fa7ee2-trusted-ca-bundle\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:04:58.675111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:58.675088 5043 scope.go:122] "RemoveContainer" containerID="4fd5f8914247df1263b73a835228c5fdd7dd1d303ce0446de641a530d30e67d6" Nov 03 20:04:59.151350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.151322 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-c6zf8-lzlj9"] Nov 03 20:04:59.155242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.155219 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-c6zf8-lzlj9"] Nov 03 20:04:59.187478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.187450 5043 generic.go:356] "Generic (PLEG): container finished" podID="db5689f3-b955-4d7c-b4b8-c203f7679fa1" containerID="e5021b488fdc03f9ab1d2eaf213e4c49ab094d230767cdc687585ff49fea00bd" exitCode=0 Nov 03 20:04:59.187573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.187522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" event={"ID":"db5689f3-b955-4d7c-b4b8-c203f7679fa1","Type":"ContainerDied","Data":"e5021b488fdc03f9ab1d2eaf213e4c49ab094d230767cdc687585ff49fea00bd"} Nov 03 20:04:59.187843 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.187827 5043 scope.go:122] "RemoveContainer" containerID="e5021b488fdc03f9ab1d2eaf213e4c49ab094d230767cdc687585ff49fea00bd" Nov 03 20:04:59.189588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.189096 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e2dbcbefeaf5f027961da4c542791e61adb71a3b3ffa4c239d98f6a35617874" Nov 03 20:04:59.189588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.189108 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:04:59.191346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:04:59.191322 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4md8j" event={"ID":"ca30aa7b-65f9-4ccb-83c5-6c555ecb08b5","Type":"ContainerStarted","Data":"756160c183b7759ad2c29c38847394c36e6747fb090a40f6ca29b1dd33c7de11"} Nov 03 20:04:59.238263 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:04:59.238229 5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"periodic-gathering-c6zf8-lzlj9\" is forbidden: User \"system:node:ip-10-0-140-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-102.ec2.internal' and this object" podUID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:05:00.197036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:00.197001 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bdwvh" event={"ID":"db5689f3-b955-4d7c-b4b8-c203f7679fa1","Type":"ContainerStarted","Data":"58c4687638d185a633daebee5de81243a553c3c8d3d090632e7261c476ea6092"} Nov 03 20:05:00.248403 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:05:00.248349 5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"periodic-gathering-c6zf8-lzlj9\" is forbidden: User \"system:node:ip-10-0-140-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-140-102.ec2.internal' and this object" podUID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" pod="openshift-insights/periodic-gathering-c6zf8-lzlj9" Nov 03 20:05:00.674207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:00.674170 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" path="/var/lib/kubelet/pods/aa7644a3-9a1a-47a8-82da-44de32fa7ee2/volumes" Nov 03 20:05:05.625556 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:05:05.625518 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:05:28.353549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:28.353497 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:35.626461 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:05:35.626425 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:05:41.314251 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:41.314212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:50.877164 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:50.877057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:50.974518 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:50.974489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:51.177228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:51.177147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:51.258725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:51.258686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:53.894309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:53.894258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:53.988320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:05:53.988289 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:05.625984 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:06:05.625949 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:06:07.455708 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:06:07.455669 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:07.556784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:06:07.556743 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:28.477305 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:06:28.477267 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:35.626414 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:06:35.626382 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:07:05.626814 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:07:05.626779 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:07:28.594916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:07:28.594837 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:35.626426 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:07:35.626388 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:07:51.250931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:07:51.250891 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:51.351701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:07:51.351668 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:05.626291 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:08:05.626254 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:08:28.733003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:28.732959 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:35.627240 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:08:35.627200 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:08:38.700619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:38.700586 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:08:38.701406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:38.701383 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:08:38.707595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:38.707575 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:08:38.708290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:38.708262 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:08:46.798386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:46.798342 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:46.906990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:08:46.906965 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:05.626402 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:09:05.626326 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:09:24.381709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:24.381667 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:24.581801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:24.581770 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:27.233895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:27.233858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:27.341178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:27.341145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:28.866235 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:28.866202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:35.626702 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:09:35.626660 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:09:38.676538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:38.676504 5043 scope.go:122] "RemoveContainer" containerID="fe3adbd709414b43e3046dfce686930becb6b485cafc2ac8e0a451c049582c99" Nov 03 20:09:49.362803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:49.362775 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:49.542092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:49.542048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:50.943581 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:50.943543 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:51.079321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:51.079287 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:52.495885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:52.495849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:56.059111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.059052 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:56.194649 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.194601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:56.209899 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.209873 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz"] Nov 03 20:09:56.210355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210313 5043 cpu_manager.go:414] "containerMap: removing container" podUID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" containerName="insights-gathering" Nov 03 20:09:56.210355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210333 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" containerName="insights-gathering" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210368 5043 cpu_manager.go:414] "containerMap: removing container" podUID="259382b1-dd55-47ac-9744-c5b956a13ce7" containerName="registry" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210376 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259382b1-dd55-47ac-9744-c5b956a13ce7" containerName="registry" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210404 5043 cpu_manager.go:414] "containerMap: removing container" podUID="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" containerName="registry" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210412 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" containerName="registry" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210484 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="259382b1-dd55-47ac-9744-c5b956a13ce7" containerName="registry" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210498 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae33a00-5cd0-4a6e-b37a-f5e6e7743bc8" containerName="registry" Nov 03 20:09:56.210552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.210508 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7644a3-9a1a-47a8-82da-44de32fa7ee2" containerName="insights-gathering" Nov 03 20:09:56.212684 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.212662 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.234041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.234016 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:09:56.244453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.244434 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz"] Nov 03 20:09:56.270508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270488 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270525 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270553 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270642 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270722 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628xx\" (UniqueName: \"kubernetes.io/projected/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-kube-api-access-628xx\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270781 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270768 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270796 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270818 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.270900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.270865 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.279708 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.279690 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tzlpr\"" Nov 03 20:09:56.303192 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.303166 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:56.345222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.345198 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr"] Nov 03 20:09:56.347562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.347546 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372158 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b84b2e-4811-4823-824d-63308ffb9739-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372194 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628xx\" (UniqueName: \"kubernetes.io/projected/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-kube-api-access-628xx\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.372291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372214 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372229 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372331 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b84b2e-4811-4823-824d-63308ffb9739-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372359 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72wq\" (UniqueName: \"kubernetes.io/projected/30b84b2e-4811-4823-824d-63308ffb9739-kube-api-access-n72wq\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372386 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372445 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372484 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372517 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b84b2e-4811-4823-824d-63308ffb9739-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.372681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372549 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.372681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372568 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.372889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372686 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.372889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372723 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.372889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372749 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.372889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372791 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372884 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372899 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.372946 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.373008 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.373028 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.373310 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.373451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.373435 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.375202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.375182 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.375538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.375514 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.382140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.382102 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-fvm2t\"" Nov 03 20:09:56.390883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.390861 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr"] Nov 03 20:09:56.395096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.395044 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.395436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.395418 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628xx\" (UniqueName: \"kubernetes.io/projected/17893f17-0fe4-4378-b6cd-f2ee0dfebd26-kube-api-access-628xx\") pod \"maas-default-gateway-openshift-default-694db55c87-dl6vz\" (UID: \"17893f17-0fe4-4378-b6cd-f2ee0dfebd26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.473467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473435 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b84b2e-4811-4823-824d-63308ffb9739-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473476 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72wq\" (UniqueName: \"kubernetes.io/projected/30b84b2e-4811-4823-824d-63308ffb9739-kube-api-access-n72wq\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473500 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473530 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473555 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473578 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b84b2e-4811-4823-824d-63308ffb9739-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473668 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b84b2e-4811-4823-824d-63308ffb9739-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473699 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473723 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473863 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473914 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.473995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.473971 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.474183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.474051 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.474307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.474290 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b84b2e-4811-4823-824d-63308ffb9739-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.475718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.475700 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b84b2e-4811-4823-824d-63308ffb9739-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.476102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.476066 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b84b2e-4811-4823-824d-63308ffb9739-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.485420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.485398 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:56.491822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.491795 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b84b2e-4811-4823-824d-63308ffb9739-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.492267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.492247 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72wq\" (UniqueName: \"kubernetes.io/projected/30b84b2e-4811-4823-824d-63308ffb9739-kube-api-access-n72wq\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr\" (UID: \"30b84b2e-4811-4823-824d-63308ffb9739\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.532692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.532669 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:09:56.599644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.599578 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:56.658787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.658760 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:09:56.843749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.843715 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz"] Nov 03 20:09:56.846537 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:09:56.846504 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17893f17_0fe4_4378_b6cd_f2ee0dfebd26.slice/crio-027f55c752a7d8173415bed16205159d1f2070300d725de2d10442222ca66a20 WatchSource:0}: Error finding container 027f55c752a7d8173415bed16205159d1f2070300d725de2d10442222ca66a20: Status 404 returned error can't find the container with id 027f55c752a7d8173415bed16205159d1f2070300d725de2d10442222ca66a20 Nov 03 20:09:56.848392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.848375 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:09:56.952373 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:56.952297 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr"] Nov 03 20:09:56.956590 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:09:56.956558 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b84b2e_4811_4823_824d_63308ffb9739.slice/crio-e992d7d023191d7c3bb763bac5c26dbd333a7d9e5ba1dec62eb47c7a10cd061a WatchSource:0}: Error finding container e992d7d023191d7c3bb763bac5c26dbd333a7d9e5ba1dec62eb47c7a10cd061a: Status 404 returned error can't find the container with id e992d7d023191d7c3bb763bac5c26dbd333a7d9e5ba1dec62eb47c7a10cd061a Nov 03 20:09:57.384202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:57.384163 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" event={"ID":"30b84b2e-4811-4823-824d-63308ffb9739","Type":"ContainerStarted","Data":"e992d7d023191d7c3bb763bac5c26dbd333a7d9e5ba1dec62eb47c7a10cd061a"} Nov 03 20:09:57.385452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:57.385420 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" event={"ID":"17893f17-0fe4-4378-b6cd-f2ee0dfebd26","Type":"ContainerStarted","Data":"027f55c752a7d8173415bed16205159d1f2070300d725de2d10442222ca66a20"} Nov 03 20:09:58.231937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.231903 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c"] Nov 03 20:09:58.235210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.235185 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.249348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.249325 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:09:58.260778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.260753 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z69q8\"" Nov 03 20:09:58.260909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.260822 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:09:58.262395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.262337 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c"] Nov 03 20:09:58.289379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.289355 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldfm\" (UniqueName: \"kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.289624 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.289504 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.289624 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.289556 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.390223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.390183 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.390583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.390242 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.390583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.390277 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldfm\" (UniqueName: \"kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.390662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.390615 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.390701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.390653 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.410382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.410338 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldfm\" (UniqueName: \"kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.551392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.551323 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:09:58.855927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:58.855894 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c"] Nov 03 20:09:59.398522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:59.398475 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" event={"ID":"771857af-be82-468e-8d5d-7c7554babc05","Type":"ContainerStarted","Data":"b20501bf4ba8d9113e6fb7487f09ba76f432a4637c925de2e511a91d5f6c824e"} Nov 03 20:09:59.884001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:59.883968 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:09:59.884147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:59.884038 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:09:59.884147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:59.884094 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:09:59.909745 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:09:59.909716 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.328530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.328488 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:00.328652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.328575 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:00.328652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.328608 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:00.404762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.404732 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" event={"ID":"17893f17-0fe4-4378-b6cd-f2ee0dfebd26","Type":"ContainerStarted","Data":"c3e7383c0e4ee94f36330991c350b8578192d2893ebef855c701af4fcd77ca3e"} Nov 03 20:10:00.461424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.461283 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podStartSLOduration=1.426742957 podStartE2EDuration="4.461264129s" totalImagesPullingTime="3.034521172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:00.458961441 +0000 UTC m=+381.925916695" watchObservedRunningTime="2026-11-03 20:10:00.461264129 +0000 UTC m=+381.928219375" Nov 03 20:10:00.533772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.533735 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:10:00.538381 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:00.538356 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:10:01.410971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:01.410930 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" event={"ID":"30b84b2e-4811-4823-824d-63308ffb9739","Type":"ContainerStarted","Data":"9880112c5c1ec323aa19f6bd7b2fc47f07a52402fb067ebf56b8a8de06c76821"} Nov 03 20:10:01.411560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:01.411000 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:10:01.412095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:01.412059 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:10:01.463806 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:01.463755 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" podStartSLOduration=2.094336469 podStartE2EDuration="5.46373667s" totalImagesPullingTime="3.369400201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.463198469 +0000 UTC m=+382.930153736" watchObservedRunningTime="2026-11-03 20:10:01.46373667 +0000 UTC m=+382.930691925" Nov 03 20:10:01.659469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:01.659436 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:10:01.664533 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:01.664475 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:10:02.416797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:02.416729 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:10:02.417604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:02.417578 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr" Nov 03 20:10:05.432963 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:05.432933 5043 generic.go:356] "Generic (PLEG): container finished" podID="771857af-be82-468e-8d5d-7c7554babc05" containerID="7efa91205e92a2ae10890999825d6a9588d36ab5eb95e14534c5ce3550461bde" exitCode=0 Nov 03 20:10:05.433359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:05.433010 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" event={"ID":"771857af-be82-468e-8d5d-7c7554babc05","Type":"ContainerDied","Data":"7efa91205e92a2ae10890999825d6a9588d36ab5eb95e14534c5ce3550461bde"} Nov 03 20:10:05.626769 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:10:05.626730 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:10:07.444378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:07.444344 5043 generic.go:356] "Generic (PLEG): container finished" podID="771857af-be82-468e-8d5d-7c7554babc05" containerID="d433a9df73485b3e2250006fabc9956b306a454d4e90c8462d056f98a80f6ca2" exitCode=0 Nov 03 20:10:07.444691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:07.444392 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" event={"ID":"771857af-be82-468e-8d5d-7c7554babc05","Type":"ContainerDied","Data":"d433a9df73485b3e2250006fabc9956b306a454d4e90c8462d056f98a80f6ca2"} Nov 03 20:10:13.473354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:13.473275 5043 generic.go:356] "Generic (PLEG): container finished" podID="771857af-be82-468e-8d5d-7c7554babc05" containerID="db99b1c36c36b40ffbbaf139bcf98979db1dece80aa2ec695c7bb07aaf69bf99" exitCode=0 Nov 03 20:10:13.473354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:13.473314 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" event={"ID":"771857af-be82-468e-8d5d-7c7554babc05","Type":"ContainerDied","Data":"db99b1c36c36b40ffbbaf139bcf98979db1dece80aa2ec695c7bb07aaf69bf99"} Nov 03 20:10:14.604047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.604023 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:10:14.740790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.740722 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm\" (UniqueName: \"kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm\") pod \"771857af-be82-468e-8d5d-7c7554babc05\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " Nov 03 20:10:14.740790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.740780 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle\") pod \"771857af-be82-468e-8d5d-7c7554babc05\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " Nov 03 20:10:14.740987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.740807 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util\") pod \"771857af-be82-468e-8d5d-7c7554babc05\" (UID: \"771857af-be82-468e-8d5d-7c7554babc05\") " Nov 03 20:10:14.741423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.741395 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle" pod "771857af-be82-468e-8d5d-7c7554babc05" (UID: "771857af-be82-468e-8d5d-7c7554babc05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:10:14.742946 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.742926 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm" pod "771857af-be82-468e-8d5d-7c7554babc05" (UID: "771857af-be82-468e-8d5d-7c7554babc05"). InnerVolumeSpecName "kube-api-access-9ldfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:14.745098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.745063 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util" pod "771857af-be82-468e-8d5d-7c7554babc05" (UID: "771857af-be82-468e-8d5d-7c7554babc05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:10:14.842402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.842383 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldfm\" (UniqueName: \"kubernetes.io/projected/771857af-be82-468e-8d5d-7c7554babc05-kube-api-access-9ldfm\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:10:14.842402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.842403 5043 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-bundle\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:10:14.842516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:14.842413 5043 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771857af-be82-468e-8d5d-7c7554babc05-util\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:10:15.484027 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:15.483990 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" event={"ID":"771857af-be82-468e-8d5d-7c7554babc05","Type":"ContainerDied","Data":"b20501bf4ba8d9113e6fb7487f09ba76f432a4637c925de2e511a91d5f6c824e"} Nov 03 20:10:15.484027 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:15.484019 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9zx5c" Nov 03 20:10:15.484247 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:15.484043 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20501bf4ba8d9113e6fb7487f09ba76f432a4637c925de2e511a91d5f6c824e" Nov 03 20:10:19.147242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:19.147211 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:20.586808 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:20.586775 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:21.508946 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.508918 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx"] Nov 03 20:10:21.509263 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509250 5043 cpu_manager.go:414] "containerMap: removing container" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="pull" Nov 03 20:10:21.509323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509264 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="pull" Nov 03 20:10:21.509323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509286 5043 cpu_manager.go:414] "containerMap: removing container" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="util" Nov 03 20:10:21.509323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509292 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="util" Nov 03 20:10:21.509323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509300 5043 cpu_manager.go:414] "containerMap: removing container" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="extract" Nov 03 20:10:21.509323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509304 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="extract" Nov 03 20:10:21.509497 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.509359 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="771857af-be82-468e-8d5d-7c7554babc05" containerName="extract" Nov 03 20:10:21.577563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.577536 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx"] Nov 03 20:10:21.577707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.577621 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.592807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.592768 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:10:21.593116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.592811 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:10:21.603257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.603239 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8pxjp\"" Nov 03 20:10:21.692318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.692295 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f3524d-c8e3-46aa-b4a2-58e059d1d66b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-djsdx\" (UID: \"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.692436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.692325 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtd4k\" (UniqueName: \"kubernetes.io/projected/a3f3524d-c8e3-46aa-b4a2-58e059d1d66b-kube-api-access-mtd4k\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-djsdx\" (UID: \"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.793255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.793193 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f3524d-c8e3-46aa-b4a2-58e059d1d66b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-djsdx\" (UID: \"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.793255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.793230 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtd4k\" (UniqueName: \"kubernetes.io/projected/a3f3524d-c8e3-46aa-b4a2-58e059d1d66b-kube-api-access-mtd4k\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-djsdx\" (UID: \"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.793530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.793509 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f3524d-c8e3-46aa-b4a2-58e059d1d66b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-djsdx\" (UID: \"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.818493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.818466 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtd4k\" (UniqueName: \"kubernetes.io/projected/a3f3524d-c8e3-46aa-b4a2-58e059d1d66b-kube-api-access-mtd4k\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-djsdx\" (UID: \"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:21.907591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:21.907572 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" Nov 03 20:10:22.242880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:22.242850 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx"] Nov 03 20:10:22.248258 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:10:22.248230 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f3524d_c8e3_46aa_b4a2_58e059d1d66b.slice/crio-fec2f67bb0394beedbf6da9068dab7ef6def81bc44c6c8ab2740074aca4a2682 WatchSource:0}: Error finding container fec2f67bb0394beedbf6da9068dab7ef6def81bc44c6c8ab2740074aca4a2682: Status 404 returned error can't find the container with id fec2f67bb0394beedbf6da9068dab7ef6def81bc44c6c8ab2740074aca4a2682 Nov 03 20:10:22.520826 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:22.520762 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" event={"ID":"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b","Type":"ContainerStarted","Data":"fec2f67bb0394beedbf6da9068dab7ef6def81bc44c6c8ab2740074aca4a2682"} Nov 03 20:10:22.947774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:22.947705 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:23.063638 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:23.063602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:24.533450 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:24.533376 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" event={"ID":"a3f3524d-c8e3-46aa-b4a2-58e059d1d66b","Type":"ContainerStarted","Data":"31cfcb03fad877647a7c9fe818d8c12335ae7e9f9be4643ce57da96cf9695712"} Nov 03 20:10:24.600011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:24.599953 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-djsdx" podStartSLOduration=1.637209705 podStartE2EDuration="3.599934126s" totalImagesPullingTime="1.962724421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:24.586816395 +0000 UTC m=+406.053771650" watchObservedRunningTime="2026-11-03 20:10:24.599934126 +0000 UTC m=+406.066889385" Nov 03 20:10:25.321313 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:25.321282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:25.429438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:25.429393 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:26.377793 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:26.377739 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:26.615500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:26.615470 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:27.306477 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.306441 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:27.444379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.444347 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq"] Nov 03 20:10:27.447261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.447238 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.485737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.485708 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:10:27.485737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.485714 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:10:27.486067 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.486055 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:10:27.486193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.486185 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:10:27.486289 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.486242 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:10:27.486334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.486303 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rsm5h\"" Nov 03 20:10:27.540971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.540948 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.541105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.540981 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b9dfcf-0e32-4489-b0af-f0a9161790a2-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.541105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.541032 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.541105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.541063 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17b9dfcf-0e32-4489-b0af-f0a9161790a2-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.541261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.541125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7htk\" (UniqueName: \"kubernetes.io/projected/17b9dfcf-0e32-4489-b0af-f0a9161790a2-kube-api-access-r7htk\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.541261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.541211 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.587817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.587794 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq"] Nov 03 20:10:27.642230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642210 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.642334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642237 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17b9dfcf-0e32-4489-b0af-f0a9161790a2-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.642334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642257 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7htk\" (UniqueName: \"kubernetes.io/projected/17b9dfcf-0e32-4489-b0af-f0a9161790a2-kube-api-access-r7htk\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.642454 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642424 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.642527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642512 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.642585 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642537 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b9dfcf-0e32-4489-b0af-f0a9161790a2-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.642680 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.642659 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17b9dfcf-0e32-4489-b0af-f0a9161790a2-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.644640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.644618 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b9dfcf-0e32-4489-b0af-f0a9161790a2-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.644765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.644744 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.644817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.644763 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.644955 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.644940 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17b9dfcf-0e32-4489-b0af-f0a9161790a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.668017 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.667992 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7htk\" (UniqueName: \"kubernetes.io/projected/17b9dfcf-0e32-4489-b0af-f0a9161790a2-kube-api-access-r7htk\") pod \"opendatahub-operator-controller-manager-6fd567655f-xh8pq\" (UID: \"17b9dfcf-0e32-4489-b0af-f0a9161790a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.689692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.689656 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:27.760917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.760883 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:27.804223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.804188 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:27.938818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:27.938737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:28.048787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:28.048757 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:28.098385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:28.098362 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq"] Nov 03 20:10:28.102130 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:10:28.102103 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b9dfcf_0e32_4489_b0af_f0a9161790a2.slice/crio-b67e34338295d158f3317314e5dadbc569cc9b93b62d6c584faac228c7da82e8 WatchSource:0}: Error finding container b67e34338295d158f3317314e5dadbc569cc9b93b62d6c584faac228c7da82e8: Status 404 returned error can't find the container with id b67e34338295d158f3317314e5dadbc569cc9b93b62d6c584faac228c7da82e8 Nov 03 20:10:28.154151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:28.154118 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:28.256118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:28.256043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:28.551264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:28.551194 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" event={"ID":"17b9dfcf-0e32-4489-b0af-f0a9161790a2","Type":"ContainerStarted","Data":"b67e34338295d158f3317314e5dadbc569cc9b93b62d6c584faac228c7da82e8"} Nov 03 20:10:28.959947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:28.959864 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.346116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:30.346067 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.446797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:30.446750 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.603663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:30.603563 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.705952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:30.705914 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.808444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:30.808393 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:31.567261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:31.567226 5043 generic.go:356] "Generic (PLEG): container finished" podID="17b9dfcf-0e32-4489-b0af-f0a9161790a2" containerID="f97d7de52c6216ceec1feb9496234b88032f8507c2c5c2fe865bf80c8f6b990f" exitCode=0 Nov 03 20:10:31.567659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:31.567312 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" event={"ID":"17b9dfcf-0e32-4489-b0af-f0a9161790a2","Type":"ContainerDied","Data":"f97d7de52c6216ceec1feb9496234b88032f8507c2c5c2fe865bf80c8f6b990f"} Nov 03 20:10:32.573527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:32.573485 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" event={"ID":"17b9dfcf-0e32-4489-b0af-f0a9161790a2","Type":"ContainerStarted","Data":"f4dc1c5b18f460c1ca4089b16a231ac931a21248842aab3bd849415928fa61c7"} Nov 03 20:10:32.573897 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:32.573568 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:32.650294 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:32.650239 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" podStartSLOduration=1.990881482 podStartE2EDuration="5.650226344s" totalImagesPullingTime="3.18396135s" totalInitContainerRuntime="475.383512ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:32.637438909 +0000 UTC m=+414.104394177" watchObservedRunningTime="2026-11-03 20:10:32.650226344 +0000 UTC m=+414.117181597" Nov 03 20:10:35.005800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:35.005704 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:35.627569 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:10:35.627522 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:10:35.749660 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:35.749619 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:35.855092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:35.855050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:38.694273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:38.694238 5043 scope.go:122] "RemoveContainer" containerID="12c93a548b3df4a3cd819d9dd2e635c1e7fac6a57b51b662d1d3440adf680056" Nov 03 20:10:39.737907 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:39.737847 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:40.174975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:40.174942 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:40.292325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:40.292292 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:43.162018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.161986 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m"] Nov 03 20:10:43.167064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.167040 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:43.194995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.194965 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 20:10:43.194995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.194971 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:10:43.194995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.194972 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:10:43.195285 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.195170 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bmpxx\"" Nov 03 20:10:43.196595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.196572 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m"] Nov 03 20:10:43.362371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.362343 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqn5d\" (UniqueName: \"kubernetes.io/projected/f84e9b68-be0d-434c-8a18-d3139cc01b6d-kube-api-access-hqn5d\") pod \"dns-operator-controller-manager-8557cbdb84-kqx9m\" (UID: \"f84e9b68-be0d-434c-8a18-d3139cc01b6d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:43.462982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.462921 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqn5d\" (UniqueName: \"kubernetes.io/projected/f84e9b68-be0d-434c-8a18-d3139cc01b6d-kube-api-access-hqn5d\") pod \"dns-operator-controller-manager-8557cbdb84-kqx9m\" (UID: \"f84e9b68-be0d-434c-8a18-d3139cc01b6d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:43.483994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.483968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqn5d\" (UniqueName: \"kubernetes.io/projected/f84e9b68-be0d-434c-8a18-d3139cc01b6d-kube-api-access-hqn5d\") pod \"dns-operator-controller-manager-8557cbdb84-kqx9m\" (UID: \"f84e9b68-be0d-434c-8a18-d3139cc01b6d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:43.581666 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.581641 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-xh8pq" Nov 03 20:10:43.778438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.778363 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:43.939410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:43.939369 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:44.072273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:44.072197 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m"] Nov 03 20:10:44.076717 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:10:44.076686 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84e9b68_be0d_434c_8a18_d3139cc01b6d.slice/crio-6f17b6c61dc914d7b2f3948053e3c87f779ec0355699219d2981e6db2df91e7e WatchSource:0}: Error finding container 6f17b6c61dc914d7b2f3948053e3c87f779ec0355699219d2981e6db2df91e7e: Status 404 returned error can't find the container with id 6f17b6c61dc914d7b2f3948053e3c87f779ec0355699219d2981e6db2df91e7e Nov 03 20:10:44.628163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:44.628125 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" event={"ID":"f84e9b68-be0d-434c-8a18-d3139cc01b6d","Type":"ContainerStarted","Data":"6f17b6c61dc914d7b2f3948053e3c87f779ec0355699219d2981e6db2df91e7e"} Nov 03 20:10:44.760668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:44.760631 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:46.182420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:46.182388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:47.645020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:47.644979 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" event={"ID":"f84e9b68-be0d-434c-8a18-d3139cc01b6d","Type":"ContainerStarted","Data":"20385ba5d19acfe09bc673124ef5c49a96913bc5d28a4d0e25d4b4e7a1527a27"} Nov 03 20:10:47.645434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:47.645222 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:47.691714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:47.691668 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" podStartSLOduration=2.100315606 podStartE2EDuration="4.691651901s" totalImagesPullingTime="2.591336295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:47.690350977 +0000 UTC m=+429.157306254" watchObservedRunningTime="2026-11-03 20:10:47.691651901 +0000 UTC m=+429.158607153" Nov 03 20:10:48.335280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.335253 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr"] Nov 03 20:10:48.337905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.337883 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:10:48.372636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.372615 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9rnzg\"" Nov 03 20:10:48.374977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.374954 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr"] Nov 03 20:10:48.399889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.399860 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjm4s\" (UniqueName: \"kubernetes.io/projected/5f971215-b5d6-469e-8374-b2beeba39c28-kube-api-access-jjm4s\") pod \"authorino-operator-7b7bb4bc77-kg5pr\" (UID: \"5f971215-b5d6-469e-8374-b2beeba39c28\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:10:48.501195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.501171 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjm4s\" (UniqueName: \"kubernetes.io/projected/5f971215-b5d6-469e-8374-b2beeba39c28-kube-api-access-jjm4s\") pod \"authorino-operator-7b7bb4bc77-kg5pr\" (UID: \"5f971215-b5d6-469e-8374-b2beeba39c28\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:10:48.519591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.519565 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjm4s\" (UniqueName: \"kubernetes.io/projected/5f971215-b5d6-469e-8374-b2beeba39c28-kube-api-access-jjm4s\") pod \"authorino-operator-7b7bb4bc77-kg5pr\" (UID: \"5f971215-b5d6-469e-8374-b2beeba39c28\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:10:48.648857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.648791 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:10:48.771457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.771423 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:48.943779 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:48.943714 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr"] Nov 03 20:10:48.948272 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:10:48.948243 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f971215_b5d6_469e_8374_b2beeba39c28.slice/crio-270b4d1654e09dac0a1eee4068e48b2c526e9c09e2c8402306a869638cc28021 WatchSource:0}: Error finding container 270b4d1654e09dac0a1eee4068e48b2c526e9c09e2c8402306a869638cc28021: Status 404 returned error can't find the container with id 270b4d1654e09dac0a1eee4068e48b2c526e9c09e2c8402306a869638cc28021 Nov 03 20:10:49.128987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:49.128948 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:49.653857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:49.653822 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" event={"ID":"5f971215-b5d6-469e-8374-b2beeba39c28","Type":"ContainerStarted","Data":"270b4d1654e09dac0a1eee4068e48b2c526e9c09e2c8402306a869638cc28021"} Nov 03 20:10:50.345771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:50.345741 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:51.290548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:51.290510 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:51.664395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:51.664359 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" event={"ID":"5f971215-b5d6-469e-8374-b2beeba39c28","Type":"ContainerStarted","Data":"aaa7d97f55fcf1ba7d559edce9e137ce6e3797bc7c0b94e8e4b015a0290c8a5c"} Nov 03 20:10:51.664594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:51.664545 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:10:51.703127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:51.703064 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" podStartSLOduration=1.883111108 podStartE2EDuration="3.703051439s" totalImagesPullingTime="1.819940331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:51.701838713 +0000 UTC m=+433.168793967" watchObservedRunningTime="2026-11-03 20:10:51.703051439 +0000 UTC m=+433.170006694" Nov 03 20:10:52.371633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:52.371605 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:52.472086 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:52.472041 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:55.578570 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.578538 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd"] Nov 03 20:10:55.582120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.582097 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.600158 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.600136 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fz5q8\"" Nov 03 20:10:55.612965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.612942 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd"] Nov 03 20:10:55.658442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.658419 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lm2pd\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.658562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.658448 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgh9w\" (UniqueName: \"kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w\") pod \"kuadrant-operator-controller-manager-678d87766f-lm2pd\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.759700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.759676 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lm2pd\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.759802 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.759708 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgh9w\" (UniqueName: \"kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w\") pod \"kuadrant-operator-controller-manager-678d87766f-lm2pd\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.760065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.760045 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lm2pd\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.778118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.778099 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgh9w\" (UniqueName: \"kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w\") pod \"kuadrant-operator-controller-manager-678d87766f-lm2pd\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:55.894696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:55.894636 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:10:56.213935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:56.213860 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd"] Nov 03 20:10:56.218794 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:10:56.218764 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5472f63c_7e21_4301_ad70_57a7c9760b82.slice/crio-0210aaa9ef14ce9c4c93eaad2f8caa095201c2eab5227b151a04a2376bc8354f WatchSource:0}: Error finding container 0210aaa9ef14ce9c4c93eaad2f8caa095201c2eab5227b151a04a2376bc8354f: Status 404 returned error can't find the container with id 0210aaa9ef14ce9c4c93eaad2f8caa095201c2eab5227b151a04a2376bc8354f Nov 03 20:10:56.566800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:56.566706 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:56.686596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:56.686561 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" event={"ID":"5472f63c-7e21-4301-ad70-57a7c9760b82","Type":"ContainerStarted","Data":"0210aaa9ef14ce9c4c93eaad2f8caa095201c2eab5227b151a04a2376bc8354f"} Nov 03 20:10:56.764691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:56.764657 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:57.822406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:57.822368 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:57.941477 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:57.941444 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:58.651129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:58.651096 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kqx9m" Nov 03 20:10:59.157622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:59.157579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:59.297890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:59.297812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:59.558654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:59.558582 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:59.981049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:10:59.981016 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.120514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.120480 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.163598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.163565 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr"] Nov 03 20:11:00.168356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.168323 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195099 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195154 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195273 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195376 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195317 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195376 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195343 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195422 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195465 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195505 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmzc\" (UniqueName: \"kubernetes.io/projected/23b2d99d-1f21-431b-9e74-8e16005ede32-kube-api-access-7cmzc\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.195624 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.195553 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b2d99d-1f21-431b-9e74-8e16005ede32-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.212090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.201313 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr"] Nov 03 20:11:00.212090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.202749 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zn8cr\"" Nov 03 20:11:00.235568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.235473 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.296274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296245 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296289 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296320 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296357 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296387 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296418 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmzc\" (UniqueName: \"kubernetes.io/projected/23b2d99d-1f21-431b-9e74-8e16005ede32-kube-api-access-7cmzc\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296465 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b2d99d-1f21-431b-9e74-8e16005ede32-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296506 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296537 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296736 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296792 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.296890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.296874 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.297135 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.297112 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.297467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.297445 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b2d99d-1f21-431b-9e74-8e16005ede32-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.299310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.299287 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.299655 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.299634 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.318554 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.318527 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b2d99d-1f21-431b-9e74-8e16005ede32-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.318968 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.318943 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmzc\" (UniqueName: \"kubernetes.io/projected/23b2d99d-1f21-431b-9e74-8e16005ede32-kube-api-access-7cmzc\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr\" (UID: \"23b2d99d-1f21-431b-9e74-8e16005ede32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.350143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.350104 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.484667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.484629 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:00.818731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.818608 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr"] Nov 03 20:11:00.823665 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:11:00.823636 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b2d99d_1f21_431b_9e74_8e16005ede32.slice/crio-a50676cd10061a90626b902d99080d7f044d952ed4ffd3009ea7d5b109d48b91 WatchSource:0}: Error finding container a50676cd10061a90626b902d99080d7f044d952ed4ffd3009ea7d5b109d48b91: Status 404 returned error can't find the container with id a50676cd10061a90626b902d99080d7f044d952ed4ffd3009ea7d5b109d48b91 Nov 03 20:11:00.826190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.826154 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:00.826309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.826247 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:00.826309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.826295 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:00.967219 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:00.967181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:01.095471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.095432 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:01.686299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.686258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:01.712342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.712302 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" event={"ID":"23b2d99d-1f21-431b-9e74-8e16005ede32","Type":"ContainerStarted","Data":"9b5e99ba7a9d07a2f0f8d370b062bdd963b28b70fc2c8bb9333e49079f8b7bf0"} Nov 03 20:11:01.712485 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.712378 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" event={"ID":"23b2d99d-1f21-431b-9e74-8e16005ede32","Type":"ContainerStarted","Data":"a50676cd10061a90626b902d99080d7f044d952ed4ffd3009ea7d5b109d48b91"} Nov 03 20:11:01.714085 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.714043 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" event={"ID":"5472f63c-7e21-4301-ad70-57a7c9760b82","Type":"ContainerStarted","Data":"f3129ebecd0fae0e647475c6bfc9b1ee4c3ad39991a29678a4c25cb4a442e23f"} Nov 03 20:11:01.714215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.714193 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:11:01.810082 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.810034 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" podStartSLOduration=1.8100218099999998 podStartE2EDuration="1.81002181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.80215348 +0000 UTC m=+443.269108754" watchObservedRunningTime="2026-11-03 20:11:01.81002181 +0000 UTC m=+443.276977064" Nov 03 20:11:01.849023 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:01.848984 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" podStartSLOduration=1.923155719 podStartE2EDuration="6.84897271s" totalImagesPullingTime="4.925816991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.847749548 +0000 UTC m=+443.314704800" watchObservedRunningTime="2026-11-03 20:11:01.84897271 +0000 UTC m=+443.315927964" Nov 03 20:11:02.485060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:02.485028 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:02.490099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:02.490047 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:02.548200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:02.548167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:02.675568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:02.675542 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kg5pr" Nov 03 20:11:02.719775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:02.719746 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:02.720732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:02.720702 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr" Nov 03 20:11:03.195880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:03.195819 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:03.324694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:03.324650 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:05.626530 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:11:05.626494 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:11:06.731743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:06.731710 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:06.833168 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:06.833046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:06.932771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:06.932736 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:07.021363 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.021279 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:07.634041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.634011 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv"] Nov 03 20:11:07.640251 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.640230 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.654254 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.654234 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:11:07.654354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.654250 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:11:07.665475 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.665451 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-qqktb\"" Nov 03 20:11:07.689806 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.689786 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:11:07.691441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.691418 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv"] Nov 03 20:11:07.760804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.760772 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7494ea-7a36-4407-9375-92ae8247bc26-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.761110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.760819 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d7494ea-7a36-4407-9375-92ae8247bc26-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.761110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.760884 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d7494ea-7a36-4407-9375-92ae8247bc26-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.761110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.760929 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zwl\" (UniqueName: \"kubernetes.io/projected/8d7494ea-7a36-4407-9375-92ae8247bc26-kube-api-access-86zwl\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.861684 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.861659 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zwl\" (UniqueName: \"kubernetes.io/projected/8d7494ea-7a36-4407-9375-92ae8247bc26-kube-api-access-86zwl\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.861777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.861688 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7494ea-7a36-4407-9375-92ae8247bc26-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.861777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.861725 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d7494ea-7a36-4407-9375-92ae8247bc26-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.861777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.861765 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d7494ea-7a36-4407-9375-92ae8247bc26-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.862170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.862149 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7494ea-7a36-4407-9375-92ae8247bc26-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.862217 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.862172 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d7494ea-7a36-4407-9375-92ae8247bc26-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.862820 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.862802 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d7494ea-7a36-4407-9375-92ae8247bc26-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.881031 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.881004 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zwl\" (UniqueName: \"kubernetes.io/projected/8d7494ea-7a36-4407-9375-92ae8247bc26-kube-api-access-86zwl\") pod \"kserve-module-controller-manager-7566cf5fc8-p4zcv\" (UID: \"8d7494ea-7a36-4407-9375-92ae8247bc26\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:07.952543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:07.952486 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:08.259372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:08.259287 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv"] Nov 03 20:11:08.263816 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:11:08.263787 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7494ea_7a36_4407_9375_92ae8247bc26.slice/crio-eeb1608542d8bba534afb25d62e226d7fb2b13b9e8659f64682f1cc17928cada WatchSource:0}: Error finding container eeb1608542d8bba534afb25d62e226d7fb2b13b9e8659f64682f1cc17928cada: Status 404 returned error can't find the container with id eeb1608542d8bba534afb25d62e226d7fb2b13b9e8659f64682f1cc17928cada Nov 03 20:11:08.753253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:08.753212 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" event={"ID":"8d7494ea-7a36-4407-9375-92ae8247bc26","Type":"ContainerStarted","Data":"eeb1608542d8bba534afb25d62e226d7fb2b13b9e8659f64682f1cc17928cada"} Nov 03 20:11:10.708670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:10.708631 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:10.764619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:10.764548 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" event={"ID":"8d7494ea-7a36-4407-9375-92ae8247bc26","Type":"ContainerStarted","Data":"48f92ffeb67ca17d4e8eb8f068c06dbd9587a6aad54a067f65524785cc856b5f"} Nov 03 20:11:10.764758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:10.764666 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:10.804014 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:10.803980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:10.819822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:10.818437 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" podStartSLOduration=1.609003714 podStartE2EDuration="3.81841978s" totalImagesPullingTime="2.209416066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:10.815500917 +0000 UTC m=+452.282456171" watchObservedRunningTime="2026-11-03 20:11:10.81841978 +0000 UTC m=+452.285375035" Nov 03 20:11:12.722363 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:12.722337 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:11:17.708293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.708261 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:17.710937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.710909 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:17.711400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.711385 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:17.725133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.725111 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hxsl2\"" Nov 03 20:11:17.725133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.725134 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:11:17.738676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.738648 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:17.837726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.837695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:17.838947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.838915 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:17.851431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.851402 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrcs\" (UniqueName: \"kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs\") pod \"limitador-limitador-74dfcfb894-j4cl6\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:17.851548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.851441 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file\") pod \"limitador-limitador-74dfcfb894-j4cl6\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:17.938707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.938666 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:17.952717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.952688 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrcs\" (UniqueName: \"kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs\") pod \"limitador-limitador-74dfcfb894-j4cl6\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:17.952852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.952719 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file\") pod \"limitador-limitador-74dfcfb894-j4cl6\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:17.953333 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.953313 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file\") pod \"limitador-limitador-74dfcfb894-j4cl6\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:17.971201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:17.971143 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrcs\" (UniqueName: \"kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs\") pod \"limitador-limitador-74dfcfb894-j4cl6\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:18.023278 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:18.023250 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:18.313580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:18.313504 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:18.319001 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:11:18.318971 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05575b23_1f88_4237_9564_a3980ef9ef54.slice/crio-92b0e2cd9f5539ba5309ae1a8d71b642735686599be674d4dc466d9009f568fb WatchSource:0}: Error finding container 92b0e2cd9f5539ba5309ae1a8d71b642735686599be674d4dc466d9009f568fb: Status 404 returned error can't find the container with id 92b0e2cd9f5539ba5309ae1a8d71b642735686599be674d4dc466d9009f568fb Nov 03 20:11:18.639656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:18.639625 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:18.744994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:18.744963 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:18.805384 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:18.805354 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" event={"ID":"05575b23-1f88-4237-9564-a3980ef9ef54","Type":"ContainerStarted","Data":"92b0e2cd9f5539ba5309ae1a8d71b642735686599be674d4dc466d9009f568fb"} Nov 03 20:11:19.810996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:19.810907 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" event={"ID":"05575b23-1f88-4237-9564-a3980ef9ef54","Type":"ContainerStarted","Data":"bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2"} Nov 03 20:11:19.810996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:19.810966 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:19.849953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:19.849910 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" podStartSLOduration=1.8137394489999998 podStartE2EDuration="2.849897345s" totalImagesPullingTime="1.036157896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:19.848542222 +0000 UTC m=+461.315497476" watchObservedRunningTime="2026-11-03 20:11:19.849897345 +0000 UTC m=+461.316852598" Nov 03 20:11:20.331275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:20.331239 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:20.434504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:20.434465 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:25.531281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:25.531239 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:25.624351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:25.624311 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:26.126253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:26.126221 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:26.227097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:26.227035 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:29.064451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:29.064417 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:30.257330 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:30.257293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:30.368280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:30.368249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:30.816320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:30.816292 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:31.736552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:31.736520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:31.771174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:31.771147 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-p4zcv" Nov 03 20:11:31.848021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:31.847989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:32.330598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:32.330564 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:32.430783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:32.430749 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:32.625110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:32.625012 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:32.625312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:32.625268 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" podUID="05575b23-1f88-4237-9564-a3980ef9ef54" containerName="limitador" containerID="cri-o://bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2" gracePeriod=30 Nov 03 20:11:32.942738 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:32.942618 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:33.043665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.043638 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:33.173641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.173616 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:33.285376 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.285308 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs\" (UniqueName: \"kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs\") pod \"05575b23-1f88-4237-9564-a3980ef9ef54\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " Nov 03 20:11:33.285515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.285374 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file\" (UniqueName: \"kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file\") pod \"05575b23-1f88-4237-9564-a3980ef9ef54\" (UID: \"05575b23-1f88-4237-9564-a3980ef9ef54\") " Nov 03 20:11:33.285783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.285761 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file" pod "05575b23-1f88-4237-9564-a3980ef9ef54" (UID: "05575b23-1f88-4237-9564-a3980ef9ef54"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:33.287433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.287413 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs" pod "05575b23-1f88-4237-9564-a3980ef9ef54" (UID: "05575b23-1f88-4237-9564-a3980ef9ef54"). InnerVolumeSpecName "kube-api-access-qcrcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:33.386341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.386310 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrcs\" (UniqueName: \"kubernetes.io/projected/05575b23-1f88-4237-9564-a3980ef9ef54-kube-api-access-qcrcs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:11:33.386341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.386336 5043 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05575b23-1f88-4237-9564-a3980ef9ef54-config-file\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:11:33.877855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.877823 5043 generic.go:356] "Generic (PLEG): container finished" podID="05575b23-1f88-4237-9564-a3980ef9ef54" containerID="bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2" exitCode=0 Nov 03 20:11:33.878020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.877880 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" Nov 03 20:11:33.878020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.877906 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" event={"ID":"05575b23-1f88-4237-9564-a3980ef9ef54","Type":"ContainerDied","Data":"bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2"} Nov 03 20:11:33.878020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.877970 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-j4cl6" event={"ID":"05575b23-1f88-4237-9564-a3980ef9ef54","Type":"ContainerDied","Data":"92b0e2cd9f5539ba5309ae1a8d71b642735686599be674d4dc466d9009f568fb"} Nov 03 20:11:33.878020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.878000 5043 scope.go:122] "RemoveContainer" containerID="bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2" Nov 03 20:11:33.887628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.887594 5043 scope.go:122] "RemoveContainer" containerID="bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2" Nov 03 20:11:33.887918 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:11:33.887895 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2\": container with ID starting with bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2 not found: ID does not exist" containerID="bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2" Nov 03 20:11:33.887994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.887926 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2"} err="failed to get container status \"bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2\": rpc error: code = NotFound desc = could not find container \"bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2\": container with ID starting with bdeb6118381d5cb0d158fd53fb73a4a4a145d9a7221ec711000982ce478901f2 not found: ID does not exist" Nov 03 20:11:33.927272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.927240 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:33.938544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:33.938519 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-j4cl6"] Nov 03 20:11:34.674983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:34.674948 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05575b23-1f88-4237-9564-a3980ef9ef54" path="/var/lib/kubelet/pods/05575b23-1f88-4237-9564-a3980ef9ef54/volumes" Nov 03 20:11:35.627303 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:11:35.627272 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:11:36.944411 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:36.944368 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:37.044891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.044861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:37.548014 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.547980 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl"] Nov 03 20:11:37.548584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.548562 5043 cpu_manager.go:414] "containerMap: removing container" podUID="05575b23-1f88-4237-9564-a3980ef9ef54" containerName="limitador" Nov 03 20:11:37.548675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.548587 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05575b23-1f88-4237-9564-a3980ef9ef54" containerName="limitador" Nov 03 20:11:37.548730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.548720 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="05575b23-1f88-4237-9564-a3980ef9ef54" containerName="limitador" Nov 03 20:11:37.551682 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.551665 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.565434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.565401 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:11:37.576751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.576730 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:11:37.578663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.578641 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl"] Nov 03 20:11:37.620951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.620924 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:37.643360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.643337 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 20:11:37.643565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.643550 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cp7lk\"" Nov 03 20:11:37.723571 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.723536 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfvp\" (UniqueName: \"kubernetes.io/projected/57422d5e-d525-4e5c-bed1-f0b3258f9c46-kube-api-access-wmfvp\") pod \"postgres-7cdbf6dc44-mbrgl\" (UID: \"57422d5e-d525-4e5c-bed1-f0b3258f9c46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.723705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.723597 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57422d5e-d525-4e5c-bed1-f0b3258f9c46-data\") pod \"postgres-7cdbf6dc44-mbrgl\" (UID: \"57422d5e-d525-4e5c-bed1-f0b3258f9c46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.824315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.824294 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57422d5e-d525-4e5c-bed1-f0b3258f9c46-data\") pod \"postgres-7cdbf6dc44-mbrgl\" (UID: \"57422d5e-d525-4e5c-bed1-f0b3258f9c46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.824425 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.824399 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfvp\" (UniqueName: \"kubernetes.io/projected/57422d5e-d525-4e5c-bed1-f0b3258f9c46-kube-api-access-wmfvp\") pod \"postgres-7cdbf6dc44-mbrgl\" (UID: \"57422d5e-d525-4e5c-bed1-f0b3258f9c46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.824730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.824713 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57422d5e-d525-4e5c-bed1-f0b3258f9c46-data\") pod \"postgres-7cdbf6dc44-mbrgl\" (UID: \"57422d5e-d525-4e5c-bed1-f0b3258f9c46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.842310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.842283 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfvp\" (UniqueName: \"kubernetes.io/projected/57422d5e-d525-4e5c-bed1-f0b3258f9c46-kube-api-access-wmfvp\") pod \"postgres-7cdbf6dc44-mbrgl\" (UID: \"57422d5e-d525-4e5c-bed1-f0b3258f9c46\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:37.863568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:37.863547 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:38.154622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:38.154547 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl"] Nov 03 20:11:38.159528 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:11:38.159501 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57422d5e_d525_4e5c_bed1_f0b3258f9c46.slice/crio-32ebc9a7e0f8d8bb13616de04b648014d33f4986427513ca9fb2847e47c92780 WatchSource:0}: Error finding container 32ebc9a7e0f8d8bb13616de04b648014d33f4986427513ca9fb2847e47c92780: Status 404 returned error can't find the container with id 32ebc9a7e0f8d8bb13616de04b648014d33f4986427513ca9fb2847e47c92780 Nov 03 20:11:38.903858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:38.903820 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" event={"ID":"57422d5e-d525-4e5c-bed1-f0b3258f9c46","Type":"ContainerStarted","Data":"32ebc9a7e0f8d8bb13616de04b648014d33f4986427513ca9fb2847e47c92780"} Nov 03 20:11:40.672903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:40.672824 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:40.756520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:40.756473 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:43.323650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:43.323629 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 20:11:43.931925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:43.931890 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" event={"ID":"57422d5e-d525-4e5c-bed1-f0b3258f9c46","Type":"ContainerStarted","Data":"296549f745665d6026c2ddfab85c2d619235529f0b6222c6b6ae29d101c0858e"} Nov 03 20:11:43.932124 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:43.932023 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:43.967816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:43.967770 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" podStartSLOduration=1.818783101 podStartE2EDuration="6.967758622s" totalImagesPullingTime="5.148975521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:43.966936656 +0000 UTC m=+485.433891909" watchObservedRunningTime="2026-11-03 20:11:43.967758622 +0000 UTC m=+485.434713875" Nov 03 20:11:49.966409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:49.966379 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mbrgl" Nov 03 20:11:56.714968 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.714888 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:56.763777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.763724 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rfjgp"] Nov 03 20:11:56.766720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.766696 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:56.793798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.793777 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:11:56.793900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.793862 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w4zzp\"" Nov 03 20:11:56.793948 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.793910 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:11:56.794238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.794223 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:11:56.799681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.799659 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rfjgp"] Nov 03 20:11:56.824246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.824223 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:56.901477 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.901436 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:56.901677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.901522 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7f9\" (UniqueName: \"kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:56.901677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.901544 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:56.935377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:56.935338 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:57.001997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.001911 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7f9\" (UniqueName: \"kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.001997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.001967 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.001997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.001999 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.004410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.004386 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.004509 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.004412 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.020647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.020620 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7f9\" (UniqueName: \"kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9\") pod \"maas-controller-579cf7c6d-rfjgp\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.074957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.074916 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:57.078807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.078787 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:11:57.169141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.169101 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66656cd6fd-7sqwf"] Nov 03 20:11:57.172621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.172599 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.208117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.208094 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-7sqwf"] Nov 03 20:11:57.304937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.304880 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpszc\" (UniqueName: \"kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.304937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.304917 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.305109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.305054 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.373306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.373271 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rfjgp"] Nov 03 20:11:57.377569 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:11:57.377546 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45666d24_741b_423c_92c8_f7b1b6c4b2fc.slice/crio-b87c4d3106509239d32f14a610b65c1b323d3e08db67cb4713ac6621fbefae5e WatchSource:0}: Error finding container b87c4d3106509239d32f14a610b65c1b323d3e08db67cb4713ac6621fbefae5e: Status 404 returned error can't find the container with id b87c4d3106509239d32f14a610b65c1b323d3e08db67cb4713ac6621fbefae5e Nov 03 20:11:57.406139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.406119 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.406237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.406155 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpszc\" (UniqueName: \"kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.406237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.406185 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.408500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.408483 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.408563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.408486 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.424658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.424637 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpszc\" (UniqueName: \"kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc\") pod \"maas-controller-66656cd6fd-7sqwf\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.484870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.484845 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:11:57.785284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:57.785246 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-7sqwf"] Nov 03 20:11:57.790021 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:11:57.789996 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49f54e4_98a5_46cc_b13c_8970fd0cc164.slice/crio-d6e8d42fe2ad26c016de0f4265ddbd9f293f64d5f2556ddffb08410082ea09c8 WatchSource:0}: Error finding container d6e8d42fe2ad26c016de0f4265ddbd9f293f64d5f2556ddffb08410082ea09c8: Status 404 returned error can't find the container with id d6e8d42fe2ad26c016de0f4265ddbd9f293f64d5f2556ddffb08410082ea09c8 Nov 03 20:11:58.012549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:58.012497 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" event={"ID":"b49f54e4-98a5-46cc-b13c-8970fd0cc164","Type":"ContainerStarted","Data":"d6e8d42fe2ad26c016de0f4265ddbd9f293f64d5f2556ddffb08410082ea09c8"} Nov 03 20:11:58.014256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:11:58.014220 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" event={"ID":"45666d24-741b-423c-92c8-f7b1b6c4b2fc","Type":"ContainerStarted","Data":"b87c4d3106509239d32f14a610b65c1b323d3e08db67cb4713ac6621fbefae5e"} Nov 03 20:12:00.675533 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:00.675506 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:00.825512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:00.825466 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:01.032871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:01.032783 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" event={"ID":"45666d24-741b-423c-92c8-f7b1b6c4b2fc","Type":"ContainerStarted","Data":"232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c"} Nov 03 20:12:01.032871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:01.032861 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:12:01.034301 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:01.034278 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" event={"ID":"b49f54e4-98a5-46cc-b13c-8970fd0cc164","Type":"ContainerStarted","Data":"d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c"} Nov 03 20:12:01.034417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:01.034402 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:12:01.079794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:01.079748 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" podStartSLOduration=2.12695495 podStartE2EDuration="5.079735563s" totalImagesPullingTime="2.952780613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:01.078013103 +0000 UTC m=+502.544968357" watchObservedRunningTime="2026-11-03 20:12:01.079735563 +0000 UTC m=+502.546690817" Nov 03 20:12:01.124382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:01.124323 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" podStartSLOduration=1.284557717 podStartE2EDuration="4.124309819s" totalImagesPullingTime="2.839752102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:01.121928087 +0000 UTC m=+502.588883355" watchObservedRunningTime="2026-11-03 20:12:01.124309819 +0000 UTC m=+502.591265074" Nov 03 20:12:03.377639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.377598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.562283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.562194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.663702 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.663618 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.699714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.699681 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kl5jp"] Nov 03 20:12:03.703865 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.703842 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.718348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.718322 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 20:12:03.726687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.726664 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9hkqg\"" Nov 03 20:12:03.732113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.732090 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kl5jp"] Nov 03 20:12:03.768711 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.768686 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/129a77fa-fdc8-4ab0-b177-6a798ca33943-cert\") pod \"kserve-controller-manager-776cc66d47-kl5jp\" (UID: \"129a77fa-fdc8-4ab0-b177-6a798ca33943\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.768818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.768743 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhblm\" (UniqueName: \"kubernetes.io/projected/129a77fa-fdc8-4ab0-b177-6a798ca33943-kube-api-access-jhblm\") pod \"kserve-controller-manager-776cc66d47-kl5jp\" (UID: \"129a77fa-fdc8-4ab0-b177-6a798ca33943\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.771738 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.771711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.785878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.785857 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4mwwp"] Nov 03 20:12:03.788757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.788741 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.812882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.812862 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4mwwp"] Nov 03 20:12:03.818452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.818432 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 20:12:03.818577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.818526 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ch5df\"" Nov 03 20:12:03.820127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.820108 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-b474r"] Nov 03 20:12:03.823092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.823060 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.845134 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.844186 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-b474r"] Nov 03 20:12:03.869868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.869841 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/129a77fa-fdc8-4ab0-b177-6a798ca33943-cert\") pod \"kserve-controller-manager-776cc66d47-kl5jp\" (UID: \"129a77fa-fdc8-4ab0-b177-6a798ca33943\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.869868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.869872 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhblm\" (UniqueName: \"kubernetes.io/projected/129a77fa-fdc8-4ab0-b177-6a798ca33943-kube-api-access-jhblm\") pod \"kserve-controller-manager-776cc66d47-kl5jp\" (UID: \"129a77fa-fdc8-4ab0-b177-6a798ca33943\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.870117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.869894 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56690b38-ee17-4993-9dde-eb4e499dd9b8-tls-certs\") pod \"model-serving-api-75bd89fdfc-4mwwp\" (UID: \"56690b38-ee17-4993-9dde-eb4e499dd9b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.870117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.869940 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-metrics-certs\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.870117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.870047 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4n2\" (UniqueName: \"kubernetes.io/projected/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-kube-api-access-6j4n2\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.870308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.870202 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-cert\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.870308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.870276 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrq4\" (UniqueName: \"kubernetes.io/projected/56690b38-ee17-4993-9dde-eb4e499dd9b8-kube-api-access-mcrq4\") pod \"model-serving-api-75bd89fdfc-4mwwp\" (UID: \"56690b38-ee17-4993-9dde-eb4e499dd9b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.872406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.872380 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/129a77fa-fdc8-4ab0-b177-6a798ca33943-cert\") pod \"kserve-controller-manager-776cc66d47-kl5jp\" (UID: \"129a77fa-fdc8-4ab0-b177-6a798ca33943\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.876462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.876437 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.887901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.887881 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhblm\" (UniqueName: \"kubernetes.io/projected/129a77fa-fdc8-4ab0-b177-6a798ca33943-kube-api-access-jhblm\") pod \"kserve-controller-manager-776cc66d47-kl5jp\" (UID: \"129a77fa-fdc8-4ab0-b177-6a798ca33943\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:03.918629 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.918571 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 20:12:03.918863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.918846 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 20:12:03.918910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.918848 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6rlhl\"" Nov 03 20:12:03.971231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.971211 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-metrics-certs\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.971342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.971260 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4n2\" (UniqueName: \"kubernetes.io/projected/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-kube-api-access-6j4n2\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.971342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.971283 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-cert\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.971342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.971315 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrq4\" (UniqueName: \"kubernetes.io/projected/56690b38-ee17-4993-9dde-eb4e499dd9b8-kube-api-access-mcrq4\") pod \"model-serving-api-75bd89fdfc-4mwwp\" (UID: \"56690b38-ee17-4993-9dde-eb4e499dd9b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.971533 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.971362 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56690b38-ee17-4993-9dde-eb4e499dd9b8-tls-certs\") pod \"model-serving-api-75bd89fdfc-4mwwp\" (UID: \"56690b38-ee17-4993-9dde-eb4e499dd9b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.974384 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.974364 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-cert\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.974470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.974364 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56690b38-ee17-4993-9dde-eb4e499dd9b8-tls-certs\") pod \"model-serving-api-75bd89fdfc-4mwwp\" (UID: \"56690b38-ee17-4993-9dde-eb4e499dd9b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.974470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.974409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-metrics-certs\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:03.989858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.989835 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrq4\" (UniqueName: \"kubernetes.io/projected/56690b38-ee17-4993-9dde-eb4e499dd9b8-kube-api-access-mcrq4\") pod \"model-serving-api-75bd89fdfc-4mwwp\" (UID: \"56690b38-ee17-4993-9dde-eb4e499dd9b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:03.990107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:03.990086 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4n2\" (UniqueName: \"kubernetes.io/projected/123dcd1e-fd84-4fff-b88d-0615c9d12cd9-kube-api-access-6j4n2\") pod \"odh-model-controller-769875db56-b474r\" (UID: \"123dcd1e-fd84-4fff-b88d-0615c9d12cd9\") " pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:04.016728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.016709 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:04.101991 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.101960 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:04.135128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.135097 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:04.162041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.159975 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:04.306737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.306702 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kl5jp"] Nov 03 20:12:04.311192 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:12:04.311167 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129a77fa_fdc8_4ab0_b177_6a798ca33943.slice/crio-924230df288cc279f3bc15f7758d715d98014ff2d07dccabb8796bb9301fa912 WatchSource:0}: Error finding container 924230df288cc279f3bc15f7758d715d98014ff2d07dccabb8796bb9301fa912: Status 404 returned error can't find the container with id 924230df288cc279f3bc15f7758d715d98014ff2d07dccabb8796bb9301fa912 Nov 03 20:12:04.396067 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.395989 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4mwwp"] Nov 03 20:12:04.399369 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:12:04.399345 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56690b38_ee17_4993_9dde_eb4e499dd9b8.slice/crio-53ed87903dccbd4bd1dd193534459af0f59925d22789ac1b745f0b63fa37fca2 WatchSource:0}: Error finding container 53ed87903dccbd4bd1dd193534459af0f59925d22789ac1b745f0b63fa37fca2: Status 404 returned error can't find the container with id 53ed87903dccbd4bd1dd193534459af0f59925d22789ac1b745f0b63fa37fca2 Nov 03 20:12:04.442177 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:04.442146 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-b474r"] Nov 03 20:12:04.445815 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:12:04.445762 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123dcd1e_fd84_4fff_b88d_0615c9d12cd9.slice/crio-3ac8d30a9fc026ae9eb691b0420ea4d8f5eaf080c6824a6bf354029400cd9613 WatchSource:0}: Error finding container 3ac8d30a9fc026ae9eb691b0420ea4d8f5eaf080c6824a6bf354029400cd9613: Status 404 returned error can't find the container with id 3ac8d30a9fc026ae9eb691b0420ea4d8f5eaf080c6824a6bf354029400cd9613 Nov 03 20:12:05.056940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.056904 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-b474r" event={"ID":"123dcd1e-fd84-4fff-b88d-0615c9d12cd9","Type":"ContainerStarted","Data":"3ac8d30a9fc026ae9eb691b0420ea4d8f5eaf080c6824a6bf354029400cd9613"} Nov 03 20:12:05.058848 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.058803 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" event={"ID":"56690b38-ee17-4993-9dde-eb4e499dd9b8","Type":"ContainerStarted","Data":"53ed87903dccbd4bd1dd193534459af0f59925d22789ac1b745f0b63fa37fca2"} Nov 03 20:12:05.061022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.060978 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" event={"ID":"129a77fa-fdc8-4ab0-b177-6a798ca33943","Type":"ContainerStarted","Data":"924230df288cc279f3bc15f7758d715d98014ff2d07dccabb8796bb9301fa912"} Nov 03 20:12:05.528940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.528894 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp"] Nov 03 20:12:05.532400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.532376 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.562288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.561001 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 20:12:05.562288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.561551 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 20:12:05.562288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.561590 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 20:12:05.562288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.561679 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-k7swk\"" Nov 03 20:12:05.564380 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.564355 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp"] Nov 03 20:12:05.589939 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.589912 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.590224 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.590205 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.590461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.590343 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dm5\" (UniqueName: \"kubernetes.io/projected/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-kube-api-access-x5dm5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.590654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.590534 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.620520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.620417 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:05.659663 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:12:05.659611 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:12:05.692237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.691747 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.692237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.691814 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.692237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.691860 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.692237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.691885 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dm5\" (UniqueName: \"kubernetes.io/projected/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-kube-api-access-x5dm5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.692237 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:12:05.692148 5043 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 20:12:05.692237 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:12:05.692234 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-proxy-source podName:5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd nodeName:}" failed. No retries permitted until 2026-11-03 20:12:06.192214417 +0000 UTC m=+507.659169667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" (UID: "5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd") : configmap "usage-tenancy-proxy-source" not found Nov 03 20:12:05.692971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.692943 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.697454 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.697426 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.712839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.712815 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dm5\" (UniqueName: \"kubernetes.io/projected/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-kube-api-access-x5dm5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:05.742434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.742388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:05.898274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:05.898239 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.003626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.003593 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.104846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.103989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.198482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.198311 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:06.199371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.199347 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rl7rp\" (UID: \"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:06.203617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.203582 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.321358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.321322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.455387 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.455306 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:06.775281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:06.775189 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp"] Nov 03 20:12:07.079421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:07.079380 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" event={"ID":"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd","Type":"ContainerStarted","Data":"be456aaa6723bb51ca1da1445f8d35ae735a4b52c86e62dafbd1265b7e03f41c"} Nov 03 20:12:07.081339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:07.081307 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" event={"ID":"56690b38-ee17-4993-9dde-eb4e499dd9b8","Type":"ContainerStarted","Data":"2364d95a086007e25623fa3cbd41213333961bf89c811774685730eac5438eab"} Nov 03 20:12:07.082160 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:07.082136 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:07.122389 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:07.121516 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" podStartSLOduration=1.743974989 podStartE2EDuration="4.121498395s" totalImagesPullingTime="2.377523406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:07.119419997 +0000 UTC m=+508.586375251" watchObservedRunningTime="2026-11-03 20:12:07.121498395 +0000 UTC m=+508.588453654" Nov 03 20:12:08.101105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:08.100104 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" event={"ID":"129a77fa-fdc8-4ab0-b177-6a798ca33943","Type":"ContainerStarted","Data":"095c561f78cb4747fbf56e5b4f106155b6357e0a940cde039e849ffb0132f9b6"} Nov 03 20:12:08.101105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:08.100190 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:08.142742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:08.142684 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" podStartSLOduration=1.935446811 podStartE2EDuration="5.142665611s" totalImagesPullingTime="3.2072188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:08.141883737 +0000 UTC m=+509.608838991" watchObservedRunningTime="2026-11-03 20:12:08.142665611 +0000 UTC m=+509.609620867" Nov 03 20:12:09.110914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:09.110354 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-b474r" event={"ID":"123dcd1e-fd84-4fff-b88d-0615c9d12cd9","Type":"ContainerStarted","Data":"4fa4a422413e5fe817334f55406bfda131245242c206e855d272aa701c94f19e"} Nov 03 20:12:09.110914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:09.110853 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:09.149286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:09.149227 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-b474r" podStartSLOduration=2.268992601 podStartE2EDuration="6.149210111s" totalImagesPullingTime="3.88021751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:09.146653439 +0000 UTC m=+510.613608693" watchObservedRunningTime="2026-11-03 20:12:09.149210111 +0000 UTC m=+510.616165366" Nov 03 20:12:12.041170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.041138 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:12:12.042007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.041990 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:12:12.107018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.106986 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rfjgp"] Nov 03 20:12:12.135215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.135174 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" podUID="45666d24-741b-423c-92c8-f7b1b6c4b2fc" containerName="manager" containerID="cri-o://232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c" gracePeriod=30 Nov 03 20:12:12.409351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.409323 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:12:12.567549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.567467 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9\" (UniqueName: \"kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9\") pod \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " Nov 03 20:12:12.567703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.567626 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs\") pod \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " Nov 03 20:12:12.567703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.567662 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert\") pod \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\" (UID: \"45666d24-741b-423c-92c8-f7b1b6c4b2fc\") " Nov 03 20:12:12.570406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.570342 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9" pod "45666d24-741b-423c-92c8-f7b1b6c4b2fc" (UID: "45666d24-741b-423c-92c8-f7b1b6c4b2fc"). InnerVolumeSpecName "kube-api-access-4v7f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:12.570540 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.570457 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert" pod "45666d24-741b-423c-92c8-f7b1b6c4b2fc" (UID: "45666d24-741b-423c-92c8-f7b1b6c4b2fc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:12.570607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.570581 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs" pod "45666d24-741b-423c-92c8-f7b1b6c4b2fc" (UID: "45666d24-741b-423c-92c8-f7b1b6c4b2fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:12.668625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.668596 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:12.668625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.668622 5043 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45666d24-741b-423c-92c8-f7b1b6c4b2fc-webhook-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:12.668812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:12.668635 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7f9\" (UniqueName: \"kubernetes.io/projected/45666d24-741b-423c-92c8-f7b1b6c4b2fc-kube-api-access-4v7f9\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:13.141656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.141624 5043 generic.go:356] "Generic (PLEG): container finished" podID="45666d24-741b-423c-92c8-f7b1b6c4b2fc" containerID="232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c" exitCode=0 Nov 03 20:12:13.142147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.141676 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" Nov 03 20:12:13.142147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.141693 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" event={"ID":"45666d24-741b-423c-92c8-f7b1b6c4b2fc","Type":"ContainerDied","Data":"232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c"} Nov 03 20:12:13.142147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.141741 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rfjgp" event={"ID":"45666d24-741b-423c-92c8-f7b1b6c4b2fc","Type":"ContainerDied","Data":"b87c4d3106509239d32f14a610b65c1b323d3e08db67cb4713ac6621fbefae5e"} Nov 03 20:12:13.142147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.141773 5043 scope.go:122] "RemoveContainer" containerID="232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c" Nov 03 20:12:13.151696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.151669 5043 scope.go:122] "RemoveContainer" containerID="232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c" Nov 03 20:12:13.151966 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:12:13.151934 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c\": container with ID starting with 232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c not found: ID does not exist" containerID="232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c" Nov 03 20:12:13.152098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.151975 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c"} err="failed to get container status \"232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c\": rpc error: code = NotFound desc = could not find container \"232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c\": container with ID starting with 232eb325e7e277c2aa70a2c126ccbc3fd6ed78c09a765507a42128979013307c not found: ID does not exist" Nov 03 20:12:13.191803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.191778 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rfjgp"] Nov 03 20:12:13.203643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:13.203613 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rfjgp"] Nov 03 20:12:14.676154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:14.676115 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45666d24-741b-423c-92c8-f7b1b6c4b2fc" path="/var/lib/kubelet/pods/45666d24-741b-423c-92c8-f7b1b6c4b2fc/volumes" Nov 03 20:12:19.115221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:19.115193 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4mwwp" Nov 03 20:12:20.182493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:20.182402 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" event={"ID":"5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd","Type":"ContainerStarted","Data":"0afacb8aa2a2f294bb30bb705ad13661765ce0afd7969ffa29379326018574d6"} Nov 03 20:12:20.182908 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:20.182583 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:20.220837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:20.220790 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" podStartSLOduration=2.133201949 podStartE2EDuration="15.22077816s" totalImagesPullingTime="13.087576211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:20.219458298 +0000 UTC m=+521.686413564" watchObservedRunningTime="2026-11-03 20:12:20.22077816 +0000 UTC m=+521.687733454" Nov 03 20:12:20.267156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:20.267131 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:21.131425 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:21.131395 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-b474r" Nov 03 20:12:29.177874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:29.177840 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:31.194839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:31.194807 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rl7rp" Nov 03 20:12:31.883819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:31.883777 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:31.983207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:31.983174 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.367864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.367824 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.468834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.468799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.567828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.567794 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.622041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.621862 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s"] Nov 03 20:12:33.622405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.622385 5043 cpu_manager.go:414] "containerMap: removing container" podUID="45666d24-741b-423c-92c8-f7b1b6c4b2fc" containerName="manager" Nov 03 20:12:33.622479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.622405 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45666d24-741b-423c-92c8-f7b1b6c4b2fc" containerName="manager" Nov 03 20:12:33.622546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.622532 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="45666d24-741b-423c-92c8-f7b1b6c4b2fc" containerName="manager" Nov 03 20:12:33.625288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.625270 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.652012 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.651991 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-49bxb\"" Nov 03 20:12:33.652012 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.651994 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:12:33.652176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.652135 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:12:33.653524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.653497 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.653639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.653565 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hng87\" (UniqueName: \"kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.653800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.653734 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.654055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.654001 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.654963 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.654940 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s"] Nov 03 20:12:33.724496 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.724465 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.754643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.754613 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hng87\" (UniqueName: \"kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.754764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.754746 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.754834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.754772 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.754834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.754821 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.755489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.755466 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.757175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.757154 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.757261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.757208 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.774613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.774582 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hng87\" (UniqueName: \"kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87\") pod \"maas-api-6d669c7fcd-ghx2s\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:33.937540 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:33.937456 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:34.231199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:34.231125 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s"] Nov 03 20:12:34.235371 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:12:34.235342 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48ac1f9_d5ce_4670_aee6_799d1b74501d.slice/crio-9e78060f9d793bcf16804ad39888d81fc8a7c0c51b598d5b49c1bc1cf1b0971a WatchSource:0}: Error finding container 9e78060f9d793bcf16804ad39888d81fc8a7c0c51b598d5b49c1bc1cf1b0971a: Status 404 returned error can't find the container with id 9e78060f9d793bcf16804ad39888d81fc8a7c0c51b598d5b49c1bc1cf1b0971a Nov 03 20:12:34.251300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:34.251273 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" event={"ID":"a48ac1f9-d5ce-4670-aee6-799d1b74501d","Type":"ContainerStarted","Data":"9e78060f9d793bcf16804ad39888d81fc8a7c0c51b598d5b49c1bc1cf1b0971a"} Nov 03 20:12:35.635158 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:12:35.635116 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:12:36.262709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:36.262679 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" event={"ID":"a48ac1f9-d5ce-4670-aee6-799d1b74501d","Type":"ContainerStarted","Data":"00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987"} Nov 03 20:12:36.262873 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:36.262760 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:36.300774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:36.300716 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" podStartSLOduration=1.8287349480000001 podStartE2EDuration="3.300699863s" totalImagesPullingTime="1.471964915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:36.299104267 +0000 UTC m=+537.766059540" watchObservedRunningTime="2026-11-03 20:12:36.300699863 +0000 UTC m=+537.767655118" Nov 03 20:12:40.125264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:40.125231 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kl5jp" Nov 03 20:12:42.272335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:42.272304 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:43.545761 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:43.545719 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:43.783154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:43.783116 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:43.988662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:43.988633 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm"] Nov 03 20:12:44.003900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.003875 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.013964 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.013935 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm"] Nov 03 20:12:44.067026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.067001 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.067208 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.067036 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.067208 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.067092 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp85\" (UniqueName: \"kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.067313 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.067205 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.168206 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.168163 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp85\" (UniqueName: \"kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.168371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.168227 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.168371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.168303 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.168371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.168337 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.169004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.168979 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.170761 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.170741 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.171039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.171019 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.188408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.188385 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp85\" (UniqueName: \"kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85\") pod \"maas-api-57c467f9cd-jg5bm\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.319200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.319133 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:44.623126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:44.622541 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm"] Nov 03 20:12:44.626612 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:12:44.626583 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319c0201_cbdc_49d2_bfec_d88a7e416e16.slice/crio-0423ecf0af37e911759d6286c2fc90c52a8b93658131e3c50f75e4652afacd30 WatchSource:0}: Error finding container 0423ecf0af37e911759d6286c2fc90c52a8b93658131e3c50f75e4652afacd30: Status 404 returned error can't find the container with id 0423ecf0af37e911759d6286c2fc90c52a8b93658131e3c50f75e4652afacd30 Nov 03 20:12:45.331456 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.331424 5043 generic.go:356] "Generic (PLEG): container finished" podID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerID="c3e7383c0e4ee94f36330991c350b8578192d2893ebef855c701af4fcd77ca3e" exitCode=0 Nov 03 20:12:45.331642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.331501 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" event={"ID":"17893f17-0fe4-4378-b6cd-f2ee0dfebd26","Type":"ContainerDied","Data":"c3e7383c0e4ee94f36330991c350b8578192d2893ebef855c701af4fcd77ca3e"} Nov 03 20:12:45.331930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.331914 5043 scope.go:122] "RemoveContainer" containerID="c3e7383c0e4ee94f36330991c350b8578192d2893ebef855c701af4fcd77ca3e" Nov 03 20:12:45.333121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.333086 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:45.333229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.333185 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:45.333282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.333232 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:45.333364 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.333342 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" event={"ID":"319c0201-cbdc-49d2-bfec-d88a7e416e16","Type":"ContainerStarted","Data":"17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531"} Nov 03 20:12:45.333409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.333386 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" event={"ID":"319c0201-cbdc-49d2-bfec-d88a7e416e16","Type":"ContainerStarted","Data":"0423ecf0af37e911759d6286c2fc90c52a8b93658131e3c50f75e4652afacd30"} Nov 03 20:12:45.333461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.333442 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:45.423855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:45.423748 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" podStartSLOduration=2.073750308 podStartE2EDuration="2.423728326s" totalImagesPullingTime="349.978018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:45.420592491 +0000 UTC m=+546.887547745" watchObservedRunningTime="2026-11-03 20:12:45.423728326 +0000 UTC m=+546.890683580" Nov 03 20:12:46.340659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:46.340617 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" event={"ID":"17893f17-0fe4-4378-b6cd-f2ee0dfebd26","Type":"ContainerStarted","Data":"6ff6e846f0046da72814d9834a53f9c640e80ae66a6aa92edde6c166f068ee41"} Nov 03 20:12:46.411418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:46.411369 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:12:46.533100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:46.533043 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:12:46.534132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:46.534108 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:12:46.534197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:46.534161 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 20:12:47.534057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:47.534021 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.534469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:47.534097 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 20:12:48.533385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:48.533353 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.533558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:48.533405 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 20:12:49.533236 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:49.533205 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.533631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:49.533256 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 20:12:50.533224 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:50.533181 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.533428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:50.533250 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 03 20:12:51.347127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.347095 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:12:51.417664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.417630 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s"] Nov 03 20:12:51.417921 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.417875 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" podUID="a48ac1f9-d5ce-4670-aee6-799d1b74501d" containerName="maas-api" containerID="cri-o://00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987" gracePeriod=30 Nov 03 20:12:51.537146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.537111 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:12:51.538465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.538448 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" Nov 03 20:12:51.719852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.719819 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:51.851472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.851408 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls\") pod \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " Nov 03 20:12:51.851472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.851451 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs\") pod \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " Nov 03 20:12:51.851674 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.851581 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87\" (UniqueName: \"kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87\") pod \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " Nov 03 20:12:51.851674 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.851612 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca\") pod \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\" (UID: \"a48ac1f9-d5ce-4670-aee6-799d1b74501d\") " Nov 03 20:12:51.852053 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.852026 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca" pod "a48ac1f9-d5ce-4670-aee6-799d1b74501d" (UID: "a48ac1f9-d5ce-4670-aee6-799d1b74501d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:51.853697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.853671 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs" pod "a48ac1f9-d5ce-4670-aee6-799d1b74501d" (UID: "a48ac1f9-d5ce-4670-aee6-799d1b74501d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:51.853778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.853721 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls" pod "a48ac1f9-d5ce-4670-aee6-799d1b74501d" (UID: "a48ac1f9-d5ce-4670-aee6-799d1b74501d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:51.853778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.853741 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87" pod "a48ac1f9-d5ce-4670-aee6-799d1b74501d" (UID: "a48ac1f9-d5ce-4670-aee6-799d1b74501d"). InnerVolumeSpecName "kube-api-access-hng87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:51.952685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.952633 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hng87\" (UniqueName: \"kubernetes.io/projected/a48ac1f9-d5ce-4670-aee6-799d1b74501d-kube-api-access-hng87\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:51.952685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.952653 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48ac1f9-d5ce-4670-aee6-799d1b74501d-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:51.952685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.952665 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:51.952685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:51.952673 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48ac1f9-d5ce-4670-aee6-799d1b74501d-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:12:52.369231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.369201 5043 generic.go:356] "Generic (PLEG): container finished" podID="a48ac1f9-d5ce-4670-aee6-799d1b74501d" containerID="00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987" exitCode=0 Nov 03 20:12:52.369403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.369241 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" Nov 03 20:12:52.369403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.369286 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" event={"ID":"a48ac1f9-d5ce-4670-aee6-799d1b74501d","Type":"ContainerDied","Data":"00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987"} Nov 03 20:12:52.369403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.369345 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s" event={"ID":"a48ac1f9-d5ce-4670-aee6-799d1b74501d","Type":"ContainerDied","Data":"9e78060f9d793bcf16804ad39888d81fc8a7c0c51b598d5b49c1bc1cf1b0971a"} Nov 03 20:12:52.369403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.369375 5043 scope.go:122] "RemoveContainer" containerID="00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987" Nov 03 20:12:52.378902 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.378884 5043 scope.go:122] "RemoveContainer" containerID="00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987" Nov 03 20:12:52.379207 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:12:52.379189 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987\": container with ID starting with 00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987 not found: ID does not exist" containerID="00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987" Nov 03 20:12:52.379262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.379216 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987"} err="failed to get container status \"00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987\": rpc error: code = NotFound desc = could not find container \"00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987\": container with ID starting with 00707f422c1f5714fb97040ec52a5ee0501a291b154ea04d6c720773ee307987 not found: ID does not exist" Nov 03 20:12:52.417399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.417375 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s"] Nov 03 20:12:52.428890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.428860 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ghx2s"] Nov 03 20:12:52.675478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:52.675402 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48ac1f9-d5ce-4670-aee6-799d1b74501d" path="/var/lib/kubelet/pods/a48ac1f9-d5ce-4670-aee6-799d1b74501d/volumes" Nov 03 20:12:56.302771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:12:56.302727 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:05.627279 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:13:05.627243 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:13:15.406205 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:15.406156 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:23.235912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:23.235836 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:29.275803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:29.275765 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:35.627830 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:13:35.627790 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:13:38.739566 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:38.739540 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:13:38.740046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:38.740028 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:13:38.746894 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:38.746866 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:13:38.747650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:38.747634 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:13:40.970818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:40.970787 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:42.880783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:42.880745 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:59.994622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:59.994588 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qhg57"] Nov 03 20:13:59.995173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:59.995043 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a48ac1f9-d5ce-4670-aee6-799d1b74501d" containerName="maas-api" Nov 03 20:13:59.995173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:59.995059 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48ac1f9-d5ce-4670-aee6-799d1b74501d" containerName="maas-api" Nov 03 20:13:59.995173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:59.995166 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48ac1f9-d5ce-4670-aee6-799d1b74501d" containerName="maas-api" Nov 03 20:13:59.997370 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:13:59.997349 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.014539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.014512 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:14:00.025996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.025971 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:14:00.026248 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.026214 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kschj\"" Nov 03 20:14:00.026344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.026328 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:14:00.028093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.028059 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qhg57"] Nov 03 20:14:00.060500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.060476 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:00.088749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.088718 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgm2s\" (UniqueName: \"kubernetes.io/projected/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-kube-api-access-wgm2s\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.088894 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.088771 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.088966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.088889 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-home\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.089035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.089015 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.089111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.089054 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.089161 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.089111 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.189896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.189865 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.190271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.189903 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm2s\" (UniqueName: \"kubernetes.io/projected/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-kube-api-access-wgm2s\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.190271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.189928 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.190271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.189969 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-home\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.190271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.190092 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.190271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.190123 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.191230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.190411 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.191230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.190541 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-home\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.191230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.191057 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.192801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.192774 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.192935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.192920 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.212163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.212132 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm2s\" (UniqueName: \"kubernetes.io/projected/e77bfbd0-7444-45e1-871b-cafaa9b9f1b1-kube-api-access-wgm2s\") pod \"echo-simulator-kserve-64b6f8f96c-qhg57\" (UID: \"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.310151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.310095 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:00.608993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.608965 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qhg57"] Nov 03 20:14:00.612393 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:00.612360 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77bfbd0_7444_45e1_871b_cafaa9b9f1b1.slice/crio-686266adb8dbb4407b4f6dc9b56449b2d83076be602238bac8587ed1dfd90aea WatchSource:0}: Error finding container 686266adb8dbb4407b4f6dc9b56449b2d83076be602238bac8587ed1dfd90aea: Status 404 returned error can't find the container with id 686266adb8dbb4407b4f6dc9b56449b2d83076be602238bac8587ed1dfd90aea Nov 03 20:14:00.707100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:00.707048 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" event={"ID":"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1","Type":"ContainerStarted","Data":"686266adb8dbb4407b4f6dc9b56449b2d83076be602238bac8587ed1dfd90aea"} Nov 03 20:14:02.099609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:02.099581 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:02.721788 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:02.721747 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" event={"ID":"e77bfbd0-7444-45e1-871b-cafaa9b9f1b1","Type":"ContainerStarted","Data":"3582f29d9d0c7f1279ed64b061ac546cc4e8285de2ff9f10f5aa175da21c841f"} Nov 03 20:14:02.761823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:02.761777 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" podStartSLOduration=2.383615839 podStartE2EDuration="3.761763575s" totalImagesPullingTime="1.378147736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:02.760790651 +0000 UTC m=+624.227745928" watchObservedRunningTime="2026-11-03 20:14:02.761763575 +0000 UTC m=+624.228718828" Nov 03 20:14:03.310590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:03.310563 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:04.316670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:04.316623 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:05.629435 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:14:05.629397 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:14:05.972176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:05.972097 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m"] Nov 03 20:14:05.975917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:05.975891 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:05.990883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:05.990860 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 20:14:06.004517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.004480 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m"] Nov 03 20:14:06.034632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.034598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:06.144574 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.144539 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.144768 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.144589 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.144768 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.144732 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.144965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.144767 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.144965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.144850 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8tl\" (UniqueName: \"kubernetes.io/projected/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-kube-api-access-ft8tl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.144965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.144928 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246242 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246298 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246438 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246509 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246537 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246630 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8tl\" (UniqueName: \"kubernetes.io/projected/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-kube-api-access-ft8tl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246680 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.246841 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.246736 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.247106 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.247054 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.248889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.248868 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.249284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.249263 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.266529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.266507 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8tl\" (UniqueName: \"kubernetes.io/projected/a324baa7-3f3d-4bb1-b80e-185ac8be80ee-kube-api-access-ft8tl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m\" (UID: \"a324baa7-3f3d-4bb1-b80e-185ac8be80ee\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.292318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.292293 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:06.597848 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.597809 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m"] Nov 03 20:14:06.605303 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:06.605273 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda324baa7_3f3d_4bb1_b80e_185ac8be80ee.slice/crio-abfc1adfd442361275b863eefa1471e0eb331596ee7ccbcc46905b396aacef5b WatchSource:0}: Error finding container abfc1adfd442361275b863eefa1471e0eb331596ee7ccbcc46905b396aacef5b: Status 404 returned error can't find the container with id abfc1adfd442361275b863eefa1471e0eb331596ee7ccbcc46905b396aacef5b Nov 03 20:14:06.743505 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:06.743472 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" event={"ID":"a324baa7-3f3d-4bb1-b80e-185ac8be80ee","Type":"ContainerStarted","Data":"abfc1adfd442361275b863eefa1471e0eb331596ee7ccbcc46905b396aacef5b"} Nov 03 20:14:08.760538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:08.760503 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" event={"ID":"a324baa7-3f3d-4bb1-b80e-185ac8be80ee","Type":"ContainerStarted","Data":"95d9ac8465082693f1c94b80fe7c3fe9574b5a7e2f884bcc07039fb552fe855b"} Nov 03 20:14:08.806705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:08.806655 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" podStartSLOduration=2.72064149 podStartE2EDuration="3.806642348s" totalImagesPullingTime="1.086000858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:08.804833614 +0000 UTC m=+630.271788887" watchObservedRunningTime="2026-11-03 20:14:08.806642348 +0000 UTC m=+630.273597601" Nov 03 20:14:09.074980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:09.074943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:09.292731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:09.292699 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:10.311146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:10.311106 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:10.324297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:10.324268 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:10.783159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:10.783130 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qhg57" Nov 03 20:14:11.074027 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.074000 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk"] Nov 03 20:14:11.077307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.077282 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.088426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.088394 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.088567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.088475 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e57714-521a-402a-b401-36d07bdd2b3b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.088567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.088512 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.088567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.088545 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dv6\" (UniqueName: \"kubernetes.io/projected/39e57714-521a-402a-b401-36d07bdd2b3b-kube-api-access-r5dv6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.088724 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.088601 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.088724 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.088630 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.091702 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.091599 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 20:14:11.105925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.105898 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk"] Nov 03 20:14:11.144743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.144699 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:11.189261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189232 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e57714-521a-402a-b401-36d07bdd2b3b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189267 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189289 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dv6\" (UniqueName: \"kubernetes.io/projected/39e57714-521a-402a-b401-36d07bdd2b3b-kube-api-access-r5dv6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189337 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189362 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189448 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189701 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.189785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.189762 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.190193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.190034 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.191805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.191779 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e57714-521a-402a-b401-36d07bdd2b3b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.192584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.192559 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e57714-521a-402a-b401-36d07bdd2b3b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.208191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.208169 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dv6\" (UniqueName: \"kubernetes.io/projected/39e57714-521a-402a-b401-36d07bdd2b3b-kube-api-access-r5dv6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fcnhk\" (UID: \"39e57714-521a-402a-b401-36d07bdd2b3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.410331 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.410247 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:11.712149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.712015 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk"] Nov 03 20:14:11.716775 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:11.716745 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e57714_521a_402a_b401_36d07bdd2b3b.slice/crio-fc1147ddecadd2ee89f2254125777a97fa450c89d040ee17ff5eb5cb101ec708 WatchSource:0}: Error finding container fc1147ddecadd2ee89f2254125777a97fa450c89d040ee17ff5eb5cb101ec708: Status 404 returned error can't find the container with id fc1147ddecadd2ee89f2254125777a97fa450c89d040ee17ff5eb5cb101ec708 Nov 03 20:14:11.776519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:11.776492 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" event={"ID":"39e57714-521a-402a-b401-36d07bdd2b3b","Type":"ContainerStarted","Data":"fc1147ddecadd2ee89f2254125777a97fa450c89d040ee17ff5eb5cb101ec708"} Nov 03 20:14:12.455240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.455046 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv"] Nov 03 20:14:12.458896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.458870 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.484627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.484590 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:14:12.486549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.486528 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv"] Nov 03 20:14:12.499950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.499925 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.500065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.499957 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.500065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.499986 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b70aee-391e-426c-b61b-c49fd5f58c73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.500196 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.500136 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.500251 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.500232 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.500309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.500267 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8vq\" (UniqueName: \"kubernetes.io/projected/45b70aee-391e-426c-b61b-c49fd5f58c73-kube-api-access-9q8vq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.517018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.516986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:12.600845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.600817 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.600969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.600855 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8vq\" (UniqueName: \"kubernetes.io/projected/45b70aee-391e-426c-b61b-c49fd5f58c73-kube-api-access-9q8vq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.600969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.600875 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.600969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.600895 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.600969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.600922 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b70aee-391e-426c-b61b-c49fd5f58c73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.600969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.600963 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.601338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.601318 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.601394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.601363 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.601470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.601449 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.603240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.603214 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b70aee-391e-426c-b61b-c49fd5f58c73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.603495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.603477 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b70aee-391e-426c-b61b-c49fd5f58c73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.619585 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.619552 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8vq\" (UniqueName: \"kubernetes.io/projected/45b70aee-391e-426c-b61b-c49fd5f58c73-kube-api-access-9q8vq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4q9mv\" (UID: \"45b70aee-391e-426c-b61b-c49fd5f58c73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.786152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.785358 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:12.786152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.785802 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" event={"ID":"39e57714-521a-402a-b401-36d07bdd2b3b","Type":"ContainerStarted","Data":"8bd11a3777362ecd6dd92e439a2cb1e358b282a986325e75813509d7f6b51e0a"} Nov 03 20:14:12.827712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:12.827344 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" podStartSLOduration=1.567686037 podStartE2EDuration="1.827324762s" totalImagesPullingTime="259.638725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:12.824565929 +0000 UTC m=+634.291521185" watchObservedRunningTime="2026-11-03 20:14:12.827324762 +0000 UTC m=+634.294280029" Nov 03 20:14:13.090628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:13.090597 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv"] Nov 03 20:14:13.095292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:13.095264 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b70aee_391e_426c_b61b_c49fd5f58c73.slice/crio-4432ce1db5acaf9f050a797b251daaf4a26481fdaf64196936bd065658fca795 WatchSource:0}: Error finding container 4432ce1db5acaf9f050a797b251daaf4a26481fdaf64196936bd065658fca795: Status 404 returned error can't find the container with id 4432ce1db5acaf9f050a797b251daaf4a26481fdaf64196936bd065658fca795 Nov 03 20:14:13.411986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:13.411366 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:13.795296 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:13.795201 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" event={"ID":"45b70aee-391e-426c-b61b-c49fd5f58c73","Type":"ContainerStarted","Data":"8adb25c3ce23310de4c929be29f1008f27d466a132da234745f94b5010e83f59"} Nov 03 20:14:13.795296 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:13.795270 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" event={"ID":"45b70aee-391e-426c-b61b-c49fd5f58c73","Type":"ContainerStarted","Data":"4432ce1db5acaf9f050a797b251daaf4a26481fdaf64196936bd065658fca795"} Nov 03 20:14:13.835589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:13.834461 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" podStartSLOduration=1.6434458699999999 podStartE2EDuration="1.834442438s" totalImagesPullingTime="190.996568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:13.831808814 +0000 UTC m=+635.298764069" watchObservedRunningTime="2026-11-03 20:14:13.834442438 +0000 UTC m=+635.301397694" Nov 03 20:14:14.786399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:14.786371 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:14.990357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:14.990324 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:16.293150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:16.293110 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:16.313489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:16.313448 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:16.833513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:16.833484 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m" Nov 03 20:14:20.733213 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:20.733177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:21.411187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:21.411148 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:21.423229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:21.423202 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:21.850557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:21.850528 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fcnhk" Nov 03 20:14:22.785943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:22.785906 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:22.798738 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:22.798716 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:22.855331 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:22.855306 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4q9mv" Nov 03 20:14:28.083321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.083281 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:28.101755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.101731 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-r554c"] Nov 03 20:14:28.105130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.105107 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.119103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.119061 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:14:28.130382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.130361 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xbqq6\"" Nov 03 20:14:28.133578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.133550 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-r554c"] Nov 03 20:14:28.137841 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.137819 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:14:28.166887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.166852 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnspl\" (UniqueName: \"kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl\") pod \"ai-gateway-controller-79567948bc-r554c\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.167166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.167143 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs\") pod \"ai-gateway-controller-79567948bc-r554c\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.224293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.224262 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:28.267801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.267772 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnspl\" (UniqueName: \"kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl\") pod \"ai-gateway-controller-79567948bc-r554c\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.267919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.267837 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs\") pod \"ai-gateway-controller-79567948bc-r554c\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.270644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.270617 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs\") pod \"ai-gateway-controller-79567948bc-r554c\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.285785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.285764 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnspl\" (UniqueName: \"kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl\") pod \"ai-gateway-controller-79567948bc-r554c\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.416994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.416931 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:28.713052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.712966 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-r554c"] Nov 03 20:14:28.716823 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:28.716798 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29eb844_b794_4526_92d5_b68b57440c65.slice/crio-69bff22c614a908cb7c37746dbddc3e40c3b090b4b0a7ee610fdf46f7c0c1153 WatchSource:0}: Error finding container 69bff22c614a908cb7c37746dbddc3e40c3b090b4b0a7ee610fdf46f7c0c1153: Status 404 returned error can't find the container with id 69bff22c614a908cb7c37746dbddc3e40c3b090b4b0a7ee610fdf46f7c0c1153 Nov 03 20:14:28.878086 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:28.878043 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" event={"ID":"a29eb844-b794-4526-92d5-b68b57440c65","Type":"ContainerStarted","Data":"69bff22c614a908cb7c37746dbddc3e40c3b090b4b0a7ee610fdf46f7c0c1153"} Nov 03 20:14:29.404318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:29.404277 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:31.903470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:31.903435 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" event={"ID":"a29eb844-b794-4526-92d5-b68b57440c65","Type":"ContainerStarted","Data":"c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e"} Nov 03 20:14:31.903949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:31.903497 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:31.955388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:31.955326 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" podStartSLOduration=1.681708082 podStartE2EDuration="3.955309256s" totalImagesPullingTime="2.273601174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:31.952782266 +0000 UTC m=+653.419737557" watchObservedRunningTime="2026-11-03 20:14:31.955309256 +0000 UTC m=+653.422264511" Nov 03 20:14:32.163026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.162945 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:32.268415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.268380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:32.278690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.278659 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-562kq"] Nov 03 20:14:32.282960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.282939 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.291042 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:14:32.291019 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463353e1_4394_41c6_8dae_59ff6f3208c1.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:14:32.291577 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:14:32.291551 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463353e1_4394_41c6_8dae_59ff6f3208c1.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:14:32.296973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.296947 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:14:32.297167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.297044 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b7h29\"" Nov 03 20:14:32.302995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.302972 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-562kq"] Nov 03 20:14:32.317614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.317589 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7"] Nov 03 20:14:32.322238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.322219 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.340299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.340273 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7"] Nov 03 20:14:32.410029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.409993 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4vx\" (UniqueName: \"kubernetes.io/projected/6077b5c3-e90d-46e9-88c7-07020c0c8e83-kube-api-access-mw4vx\") pod \"payload-pre-processing-7b98bb678c-tdvk7\" (UID: \"6077b5c3-e90d-46e9-88c7-07020c0c8e83\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.410228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.410040 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6077b5c3-e90d-46e9-88c7-07020c0c8e83-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tdvk7\" (UID: \"6077b5c3-e90d-46e9-88c7-07020c0c8e83\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.410228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.410186 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6822k\" (UniqueName: \"kubernetes.io/projected/463353e1-4394-41c6-8dae-59ff6f3208c1-kube-api-access-6822k\") pod \"payload-processing-7bb9489676-562kq\" (UID: \"463353e1-4394-41c6-8dae-59ff6f3208c1\") " pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.410315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.410294 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463353e1-4394-41c6-8dae-59ff6f3208c1-plugins-config-volume\") pod \"payload-processing-7bb9489676-562kq\" (UID: \"463353e1-4394-41c6-8dae-59ff6f3208c1\") " pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.511063 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.510976 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463353e1-4394-41c6-8dae-59ff6f3208c1-plugins-config-volume\") pod \"payload-processing-7bb9489676-562kq\" (UID: \"463353e1-4394-41c6-8dae-59ff6f3208c1\") " pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.511256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.511062 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4vx\" (UniqueName: \"kubernetes.io/projected/6077b5c3-e90d-46e9-88c7-07020c0c8e83-kube-api-access-mw4vx\") pod \"payload-pre-processing-7b98bb678c-tdvk7\" (UID: \"6077b5c3-e90d-46e9-88c7-07020c0c8e83\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.511256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.511125 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6077b5c3-e90d-46e9-88c7-07020c0c8e83-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tdvk7\" (UID: \"6077b5c3-e90d-46e9-88c7-07020c0c8e83\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.511256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.511207 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6822k\" (UniqueName: \"kubernetes.io/projected/463353e1-4394-41c6-8dae-59ff6f3208c1-kube-api-access-6822k\") pod \"payload-processing-7bb9489676-562kq\" (UID: \"463353e1-4394-41c6-8dae-59ff6f3208c1\") " pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.511988 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.511960 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6077b5c3-e90d-46e9-88c7-07020c0c8e83-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tdvk7\" (UID: \"6077b5c3-e90d-46e9-88c7-07020c0c8e83\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.512134 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.511968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463353e1-4394-41c6-8dae-59ff6f3208c1-plugins-config-volume\") pod \"payload-processing-7bb9489676-562kq\" (UID: \"463353e1-4394-41c6-8dae-59ff6f3208c1\") " pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.529272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.529243 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6822k\" (UniqueName: \"kubernetes.io/projected/463353e1-4394-41c6-8dae-59ff6f3208c1-kube-api-access-6822k\") pod \"payload-processing-7bb9489676-562kq\" (UID: \"463353e1-4394-41c6-8dae-59ff6f3208c1\") " pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.529383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.529313 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4vx\" (UniqueName: \"kubernetes.io/projected/6077b5c3-e90d-46e9-88c7-07020c0c8e83-kube-api-access-mw4vx\") pod \"payload-pre-processing-7b98bb678c-tdvk7\" (UID: \"6077b5c3-e90d-46e9-88c7-07020c0c8e83\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.601215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.601185 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:32.636438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.636397 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:32.902044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.902009 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-562kq"] Nov 03 20:14:32.907132 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:32.907099 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463353e1_4394_41c6_8dae_59ff6f3208c1.slice/crio-a35d4a6809c6b61f428bbcafb68f81c586981f7b216399e39e43b802f1c11bc9 WatchSource:0}: Error finding container a35d4a6809c6b61f428bbcafb68f81c586981f7b216399e39e43b802f1c11bc9: Status 404 returned error can't find the container with id a35d4a6809c6b61f428bbcafb68f81c586981f7b216399e39e43b802f1c11bc9 Nov 03 20:14:32.955597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:32.955569 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7"] Nov 03 20:14:32.960087 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:32.960045 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6077b5c3_e90d_46e9_88c7_07020c0c8e83.slice/crio-a34b197724202f60a0a952fd1d5a3e5258c107aff6a048fc6bcd3c05c4969c28 WatchSource:0}: Error finding container a34b197724202f60a0a952fd1d5a3e5258c107aff6a048fc6bcd3c05c4969c28: Status 404 returned error can't find the container with id a34b197724202f60a0a952fd1d5a3e5258c107aff6a048fc6bcd3c05c4969c28 Nov 03 20:14:33.922509 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:33.922429 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-562kq" event={"ID":"463353e1-4394-41c6-8dae-59ff6f3208c1","Type":"ContainerStarted","Data":"a35d4a6809c6b61f428bbcafb68f81c586981f7b216399e39e43b802f1c11bc9"} Nov 03 20:14:33.925204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:33.925160 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" event={"ID":"6077b5c3-e90d-46e9-88c7-07020c0c8e83","Type":"ContainerStarted","Data":"a34b197724202f60a0a952fd1d5a3e5258c107aff6a048fc6bcd3c05c4969c28"} Nov 03 20:14:35.628556 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:14:35.628458 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:14:35.940744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:35.940665 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-562kq" event={"ID":"463353e1-4394-41c6-8dae-59ff6f3208c1","Type":"ContainerStarted","Data":"9b969f9543a85a232eb3101bce8b1fff4e8e09062060c44c3ce6d135a27b67f8"} Nov 03 20:14:35.940744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:35.940740 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:35.942290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:35.942265 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" event={"ID":"6077b5c3-e90d-46e9-88c7-07020c0c8e83","Type":"ContainerStarted","Data":"f47b60ae6eecbe604e803d2504e440ff76c611667ead03b6511c26a6ca0dd133"} Nov 03 20:14:35.942428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:35.942388 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:35.978979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:35.978941 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-562kq" podStartSLOduration=1.5234373159999999 podStartE2EDuration="3.978929306s" totalImagesPullingTime="2.45549199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:35.977994621 +0000 UTC m=+657.444949928" watchObservedRunningTime="2026-11-03 20:14:35.978929306 +0000 UTC m=+657.445884554" Nov 03 20:14:36.014120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:36.014081 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" podStartSLOduration=1.537859127 podStartE2EDuration="4.014061142s" totalImagesPullingTime="2.476202015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:36.012605774 +0000 UTC m=+657.479561039" watchObservedRunningTime="2026-11-03 20:14:36.014061142 +0000 UTC m=+657.481016396" Nov 03 20:14:42.072657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.072625 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8"] Nov 03 20:14:42.079499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.079477 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.104436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.104410 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8"] Nov 03 20:14:42.203163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203126 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203195 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203244 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203288 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2044f915-8f34-4c29-ac6b-754910c41df8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203321 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203531 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203470 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203531 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203499 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2044f915-8f34-4c29-ac6b-754910c41df8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203531 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2044f915-8f34-4c29-ac6b-754910c41df8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.203625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.203565 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qsj\" (UniqueName: \"kubernetes.io/projected/2044f915-8f34-4c29-ac6b-754910c41df8-kube-api-access-j4qsj\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304604 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304634 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2044f915-8f34-4c29-ac6b-754910c41df8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304651 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2044f915-8f34-4c29-ac6b-754910c41df8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304676 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qsj\" (UniqueName: \"kubernetes.io/projected/2044f915-8f34-4c29-ac6b-754910c41df8-kube-api-access-j4qsj\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304705 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304734 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304772 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304805 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2044f915-8f34-4c29-ac6b-754910c41df8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.304997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.304831 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.305198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.305138 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.305274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.305249 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.305334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.305297 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.305334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.305322 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.305459 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.305440 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2044f915-8f34-4c29-ac6b-754910c41df8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.306958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.306934 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2044f915-8f34-4c29-ac6b-754910c41df8-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.307328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.307311 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2044f915-8f34-4c29-ac6b-754910c41df8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.322517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.322496 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2044f915-8f34-4c29-ac6b-754910c41df8-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.322656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.322639 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qsj\" (UniqueName: \"kubernetes.io/projected/2044f915-8f34-4c29-ac6b-754910c41df8-kube-api-access-j4qsj\") pod \"maas-default-gateway-openshift-default-694db55c87-5mbh8\" (UID: \"2044f915-8f34-4c29-ac6b-754910c41df8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.394662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.394637 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:42.705242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.705174 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8"] Nov 03 20:14:42.711677 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:14:42.711652 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044f915_8f34_4c29_ac6b_754910c41df8.slice/crio-2311a639389c4293663b0d4f8c2f0b919a6cb7d175f450b6bb49fb70b2aa2b28 WatchSource:0}: Error finding container 2311a639389c4293663b0d4f8c2f0b919a6cb7d175f450b6bb49fb70b2aa2b28: Status 404 returned error can't find the container with id 2311a639389c4293663b0d4f8c2f0b919a6cb7d175f450b6bb49fb70b2aa2b28 Nov 03 20:14:42.713736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.713709 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:42.713786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.713770 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:42.713822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.713800 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:42.911576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.911550 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:42.982358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.982263 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" event={"ID":"2044f915-8f34-4c29-ac6b-754910c41df8","Type":"ContainerStarted","Data":"9f44f0e04dc0ff56baefba9dcd228609074d81ae11ed7d115ff0d656e5c4e798"} Nov 03 20:14:42.982358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:42.982326 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" event={"ID":"2044f915-8f34-4c29-ac6b-754910c41df8","Type":"ContainerStarted","Data":"2311a639389c4293663b0d4f8c2f0b919a6cb7d175f450b6bb49fb70b2aa2b28"} Nov 03 20:14:43.025569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:43.025521 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podStartSLOduration=1.025506295 podStartE2EDuration="1.025506295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:43.022726549 +0000 UTC m=+664.489681822" watchObservedRunningTime="2026-11-03 20:14:43.025506295 +0000 UTC m=+664.492461549" Nov 03 20:14:43.399332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:43.399248 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:43.404215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:43.404184 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:43.404382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:43.404247 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:44.395405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:44.395368 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.395701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:44.395671 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:45.397096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:45.396422 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:45.397096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:45.396493 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:46.398094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:46.397361 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:46.398094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:46.397430 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:46.950601 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:46.950574 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tdvk7" Nov 03 20:14:46.950862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:46.950842 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-562kq" Nov 03 20:14:47.400095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:47.397429 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:47.400095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:47.397490 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:48.403469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:48.403425 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:48.403862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:48.403499 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:49.395369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:49.395333 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:14:49.395519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:49.395394 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 20:14:50.924960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:50.924867 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:50.925412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:50.925360 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:50.926093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:50.926050 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" Nov 03 20:14:58.632985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.632951 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-r554c"] Nov 03 20:14:58.633464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.633194 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" podUID="a29eb844-b794-4526-92d5-b68b57440c65" containerName="manager" containerID="cri-o://c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e" gracePeriod=30 Nov 03 20:14:58.892029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.891965 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:58.948872 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.948834 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs\") pod \"a29eb844-b794-4526-92d5-b68b57440c65\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " Nov 03 20:14:58.948992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.948910 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl\" (UniqueName: \"kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl\") pod \"a29eb844-b794-4526-92d5-b68b57440c65\" (UID: \"a29eb844-b794-4526-92d5-b68b57440c65\") " Nov 03 20:14:58.951520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.951496 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs" pod "a29eb844-b794-4526-92d5-b68b57440c65" (UID: "a29eb844-b794-4526-92d5-b68b57440c65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:58.951632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:58.951543 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl" pod "a29eb844-b794-4526-92d5-b68b57440c65" (UID: "a29eb844-b794-4526-92d5-b68b57440c65"). InnerVolumeSpecName "kube-api-access-mnspl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:59.050769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.050737 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29eb844-b794-4526-92d5-b68b57440c65-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:14:59.050769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.050764 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnspl\" (UniqueName: \"kubernetes.io/projected/a29eb844-b794-4526-92d5-b68b57440c65-kube-api-access-mnspl\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:14:59.094414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.094383 5043 generic.go:356] "Generic (PLEG): container finished" podID="a29eb844-b794-4526-92d5-b68b57440c65" containerID="c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e" exitCode=0 Nov 03 20:14:59.094534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.094426 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" Nov 03 20:14:59.094534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.094461 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" event={"ID":"a29eb844-b794-4526-92d5-b68b57440c65","Type":"ContainerDied","Data":"c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e"} Nov 03 20:14:59.094534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.094512 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-r554c" event={"ID":"a29eb844-b794-4526-92d5-b68b57440c65","Type":"ContainerDied","Data":"69bff22c614a908cb7c37746dbddc3e40c3b090b4b0a7ee610fdf46f7c0c1153"} Nov 03 20:14:59.094534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.094534 5043 scope.go:122] "RemoveContainer" containerID="c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e" Nov 03 20:14:59.105101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.105059 5043 scope.go:122] "RemoveContainer" containerID="c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e" Nov 03 20:14:59.105354 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:14:59.105332 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e\": container with ID starting with c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e not found: ID does not exist" containerID="c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e" Nov 03 20:14:59.105423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.105361 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e"} err="failed to get container status \"c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e\": rpc error: code = NotFound desc = could not find container \"c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e\": container with ID starting with c4713822ebe9eb048dbf28d098e4e20c302b5d1a8ba14591deb2f91133fcae8e not found: ID does not exist" Nov 03 20:14:59.141865 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.141836 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-r554c"] Nov 03 20:14:59.144114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:14:59.144030 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-r554c"] Nov 03 20:15:00.166973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:00.166936 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:00.627458 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:00.627420 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:00.676381 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:00.676350 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29eb844-b794-4526-92d5-b68b57440c65" path="/var/lib/kubelet/pods/a29eb844-b794-4526-92d5-b68b57440c65/volumes" Nov 03 20:15:02.125807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:02.125775 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:02.226935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:02.226896 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:03.534981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:03.534937 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:03.641818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:03.641787 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:05.626679 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:15:05.626637 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:15:24.654529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:24.654487 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:24.793805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:24.793767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.126325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.126292 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.243090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.236695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.350204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.350170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.462437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.462278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.579480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.579440 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.682937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.682901 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.782644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.782471 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.889997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.889933 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.986150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:25.986116 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:26.088528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:26.088228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:26.187230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:26.187197 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.371953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:29.371920 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.474423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:29.474389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.575390 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:29.575356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.677949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:29.677873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.790104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:29.790055 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.908725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:29.908667 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.035054 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.034883 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.136249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.136219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.248060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.248026 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.354043 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.354009 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.467494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.467461 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.688472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.688390 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.818040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:30.817995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:31.403451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.403415 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:31.506858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.506824 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:31.569799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.569773 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk"] Nov 03 20:15:31.570309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.570295 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a29eb844-b794-4526-92d5-b68b57440c65" containerName="manager" Nov 03 20:15:31.570369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.570311 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29eb844-b794-4526-92d5-b68b57440c65" containerName="manager" Nov 03 20:15:31.570405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.570401 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29eb844-b794-4526-92d5-b68b57440c65" containerName="manager" Nov 03 20:15:31.572902 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.572887 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.586797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.586772 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-fd01ad\"" Nov 03 20:15:31.600564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.600538 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk"] Nov 03 20:15:31.642257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.642226 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-fd01ad-dockercfg-cqccf\"" Nov 03 20:15:31.655053 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.654978 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.655195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.655129 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9497\" (UniqueName: \"kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497\") pod \"payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.661481 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.661455 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th"] Nov 03 20:15:31.664367 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.664353 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.688182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.688161 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th"] Nov 03 20:15:31.755817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.755782 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.755978 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.755831 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkh42\" (UniqueName: \"kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42\") pod \"payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.755978 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.755884 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.755978 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.755919 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9497\" (UniqueName: \"kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497\") pod \"payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.756552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.756530 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.774424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.774399 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9497\" (UniqueName: \"kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497\") pod \"payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.856913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.856881 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.857088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.857002 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkh42\" (UniqueName: \"kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42\") pod \"payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.857566 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.857546 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.874784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.874755 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkh42\" (UniqueName: \"kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42\") pod \"payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:31.885602 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.885582 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:31.976402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:31.976373 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:32.179790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.179746 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk"] Nov 03 20:15:32.184387 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:32.184355 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc367b1f5_858e_4c27_bcfc_56af079f456b.slice/crio-748ce6c724a3d7b204ebd29f1f3d450ca13bd4cc3ea85e72cefaf75866a9b8f7 WatchSource:0}: Error finding container 748ce6c724a3d7b204ebd29f1f3d450ca13bd4cc3ea85e72cefaf75866a9b8f7: Status 404 returned error can't find the container with id 748ce6c724a3d7b204ebd29f1f3d450ca13bd4cc3ea85e72cefaf75866a9b8f7 Nov 03 20:15:32.274708 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.274681 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th"] Nov 03 20:15:32.277944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.277919 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" event={"ID":"c367b1f5-858e-4c27-bcfc-56af079f456b","Type":"ContainerStarted","Data":"b31767f6f593d6cf34b2e64cff7b7ff60df272dc5360a9748aed242e330bb511"} Nov 03 20:15:32.278036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.277964 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" event={"ID":"c367b1f5-858e-4c27-bcfc-56af079f456b","Type":"ContainerStarted","Data":"748ce6c724a3d7b204ebd29f1f3d450ca13bd4cc3ea85e72cefaf75866a9b8f7"} Nov 03 20:15:32.278036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.277993 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:32.278381 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:32.278346 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf935dcc5_d66f_4c4e_8a2f_f844f2c3dd4c.slice/crio-cf5ee026219243d6b0ad8b9f14afb982d0297dfa9d6971e07ed92b004176d60a WatchSource:0}: Error finding container cf5ee026219243d6b0ad8b9f14afb982d0297dfa9d6971e07ed92b004176d60a: Status 404 returned error can't find the container with id cf5ee026219243d6b0ad8b9f14afb982d0297dfa9d6971e07ed92b004176d60a Nov 03 20:15:32.314786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.314740 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" podStartSLOduration=1.314723868 podStartE2EDuration="1.314723868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:32.313341707 +0000 UTC m=+713.780296962" watchObservedRunningTime="2026-11-03 20:15:32.314723868 +0000 UTC m=+713.781679127" Nov 03 20:15:32.792688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.792658 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn"] Nov 03 20:15:32.795692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.795675 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.813029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.813005 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-482722-serving-cert\"" Nov 03 20:15:32.828016 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.827990 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn"] Nov 03 20:15:32.865382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.865339 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.865545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.865481 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.865635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.865587 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.865696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.865644 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhmm\" (UniqueName: \"kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.872048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.872014 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:32.966864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.966832 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhmm\" (UniqueName: \"kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.967008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.966991 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.967060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.967036 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.967151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.967120 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.967829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.967802 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.969923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.969900 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.972225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.972196 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:32.984109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.978302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:32.985724 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:32.985701 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhmm\" (UniqueName: \"kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm\") pod \"maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:33.077339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.077303 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:33.108307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.108285 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:33.142869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.142839 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj"] Nov 03 20:15:33.148240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.148221 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.173558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.170370 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj"] Nov 03 20:15:33.173558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.170847 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.173558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.171046 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-9e1bcb\"" Nov 03 20:15:33.173558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.171103 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-9e1bcb-dockercfg-dmbxm\"" Nov 03 20:15:33.173558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.171172 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcnc\" (UniqueName: \"kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc\") pod \"payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.182384 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.182355 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:33.206993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.206970 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr"] Nov 03 20:15:33.210166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.210148 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.236114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.232726 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr"] Nov 03 20:15:33.272383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.272346 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.272586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.272444 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcnc\" (UniqueName: \"kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc\") pod \"payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.272586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.272551 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bks47\" (UniqueName: \"kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47\") pod \"payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.272684 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.272603 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.275005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.274974 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.287565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.287529 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" event={"ID":"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c","Type":"ContainerStarted","Data":"c276c09521edb817935aea7eadd0e4345a4b4c8183c209328fd0b9d9944ec001"} Nov 03 20:15:33.287743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.287588 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" event={"ID":"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c","Type":"ContainerStarted","Data":"cf5ee026219243d6b0ad8b9f14afb982d0297dfa9d6971e07ed92b004176d60a"} Nov 03 20:15:33.287819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.287764 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:33.293156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.293127 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcnc\" (UniqueName: \"kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc\") pod \"payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.326582 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.326531 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" podStartSLOduration=2.326516863 podStartE2EDuration="2.326516863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:33.324435657 +0000 UTC m=+714.791390911" watchObservedRunningTime="2026-11-03 20:15:33.326516863 +0000 UTC m=+714.793472116" Nov 03 20:15:33.373356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.373290 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bks47\" (UniqueName: \"kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47\") pod \"payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.373484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.373372 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.374058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.374033 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.391726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.391697 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bks47\" (UniqueName: \"kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47\") pod \"payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.408794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.408773 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn"] Nov 03 20:15:33.412852 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:33.412821 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344ac58a_c539_4a11_85f5_1c285fd570bf.slice/crio-738f2471120339ed3f19ea2901e4a1ad58579b848414fabd41dd9c7271e1cf51 WatchSource:0}: Error finding container 738f2471120339ed3f19ea2901e4a1ad58579b848414fabd41dd9c7271e1cf51: Status 404 returned error can't find the container with id 738f2471120339ed3f19ea2901e4a1ad58579b848414fabd41dd9c7271e1cf51 Nov 03 20:15:33.414398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.414380 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:15:33.464815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.464784 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:33.527275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.527248 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:33.764938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.764848 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj"] Nov 03 20:15:33.768993 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:33.768963 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64147b6_3d77_485d_979f_09b3cdbdfe49.slice/crio-4139b3f83f80b0505de5995bfa6bce9aaf31f37c14f33ecbf182868252f47874 WatchSource:0}: Error finding container 4139b3f83f80b0505de5995bfa6bce9aaf31f37c14f33ecbf182868252f47874: Status 404 returned error can't find the container with id 4139b3f83f80b0505de5995bfa6bce9aaf31f37c14f33ecbf182868252f47874 Nov 03 20:15:33.825546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.825515 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr"] Nov 03 20:15:33.829976 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:33.829947 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafe64f1_b7d6_4819_ac96_1d6851b30dde.slice/crio-0786ac20435ce5a002850e7e9915e182c4b9bdede02d59cdf34db66990db04c7 WatchSource:0}: Error finding container 0786ac20435ce5a002850e7e9915e182c4b9bdede02d59cdf34db66990db04c7: Status 404 returned error can't find the container with id 0786ac20435ce5a002850e7e9915e182c4b9bdede02d59cdf34db66990db04c7 Nov 03 20:15:33.847891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.847844 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:33.943974 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.943948 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f"] Nov 03 20:15:33.947862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.947844 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:33.954359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.954336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:33.966364 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.966252 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-482722\"" Nov 03 20:15:33.966521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.966377 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-482722-dockercfg-5dh9d\"" Nov 03 20:15:33.967064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.967040 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f"] Nov 03 20:15:33.980175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.980125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqbj\" (UniqueName: \"kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj\") pod \"payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:33.980323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:33.980197 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:34.001572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.001536 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j"] Nov 03 20:15:34.005084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.005057 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.024047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.023961 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j"] Nov 03 20:15:34.081037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.080976 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqbj\" (UniqueName: \"kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj\") pod \"payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:34.081037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.081031 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:34.081296 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.081083 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.081296 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.081247 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhlb\" (UniqueName: \"kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb\") pod \"payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.082618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.082582 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:34.102323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.102292 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqbj\" (UniqueName: \"kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj\") pod \"payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:34.182001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.181970 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.182220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.182185 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhlb\" (UniqueName: \"kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb\") pod \"payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.182775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.182757 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.200144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.200120 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhlb\" (UniqueName: \"kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb\") pod \"payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.261030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.261003 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:34.294937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.294870 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" event={"ID":"bafe64f1-b7d6-4819-ac96-1d6851b30dde","Type":"ContainerStarted","Data":"848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe"} Nov 03 20:15:34.294937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.294924 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" event={"ID":"bafe64f1-b7d6-4819-ac96-1d6851b30dde","Type":"ContainerStarted","Data":"0786ac20435ce5a002850e7e9915e182c4b9bdede02d59cdf34db66990db04c7"} Nov 03 20:15:34.295183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.294969 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:34.297040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.297009 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" event={"ID":"344ac58a-c539-4a11-85f5-1c285fd570bf","Type":"ContainerStarted","Data":"a3a9f3b53391022488968a2a43da87e2c8b310067d983ed50120f44874729d73"} Nov 03 20:15:34.297210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.297065 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" event={"ID":"344ac58a-c539-4a11-85f5-1c285fd570bf","Type":"ContainerStarted","Data":"738f2471120339ed3f19ea2901e4a1ad58579b848414fabd41dd9c7271e1cf51"} Nov 03 20:15:34.297210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.297136 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:34.298945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.298921 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" event={"ID":"a64147b6-3d77-485d-979f-09b3cdbdfe49","Type":"ContainerStarted","Data":"36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb"} Nov 03 20:15:34.299103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.298964 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" event={"ID":"a64147b6-3d77-485d-979f-09b3cdbdfe49","Type":"ContainerStarted","Data":"4139b3f83f80b0505de5995bfa6bce9aaf31f37c14f33ecbf182868252f47874"} Nov 03 20:15:34.324131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.317461 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:34.332876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.332828 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" podStartSLOduration=1.332814701 podStartE2EDuration="1.332814701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:34.330565192 +0000 UTC m=+715.797520447" watchObservedRunningTime="2026-11-03 20:15:34.332814701 +0000 UTC m=+715.799769954" Nov 03 20:15:34.367228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.367175 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" podStartSLOduration=1.36715868 podStartE2EDuration="1.36715868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:34.366770627 +0000 UTC m=+715.833725882" watchObservedRunningTime="2026-11-03 20:15:34.36715868 +0000 UTC m=+715.834113934" Nov 03 20:15:34.412945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.412897 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" podStartSLOduration=1.950718743 podStartE2EDuration="2.412883466s" totalImagesPullingTime="462.164723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:34.411844849 +0000 UTC m=+715.878800122" watchObservedRunningTime="2026-11-03 20:15:34.412883466 +0000 UTC m=+715.879838723" Nov 03 20:15:34.555949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.555858 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f"] Nov 03 20:15:34.562759 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:34.562722 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdb90f0_5685_43db_892b_d7f803178ad1.slice/crio-97262d210418faa544750598fee6b917542b0502d9c8bb0a1408273475e80732 WatchSource:0}: Error finding container 97262d210418faa544750598fee6b917542b0502d9c8bb0a1408273475e80732: Status 404 returned error can't find the container with id 97262d210418faa544750598fee6b917542b0502d9c8bb0a1408273475e80732 Nov 03 20:15:34.631801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.631775 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j"] Nov 03 20:15:34.639347 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:34.639316 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261cb760_7907_4d77_acf3_2f2cd927d303.slice/crio-cafccfdf8f40bb9f14a639bdc1faf17dc407e004a53cebd79808627d40fab342 WatchSource:0}: Error finding container cafccfdf8f40bb9f14a639bdc1faf17dc407e004a53cebd79808627d40fab342: Status 404 returned error can't find the container with id cafccfdf8f40bb9f14a639bdc1faf17dc407e004a53cebd79808627d40fab342 Nov 03 20:15:34.708435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.708407 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:34.769856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.769822 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm"] Nov 03 20:15:34.773107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.773082 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.787116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.787088 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9973d9-serving-cert\"" Nov 03 20:15:34.795621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.795591 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.795762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.795628 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2d5\" (UniqueName: \"kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.795762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.795746 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.795879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.795856 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.798402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.798378 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm"] Nov 03 20:15:34.823022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.822864 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:34.863604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.863577 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn"] Nov 03 20:15:34.866719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.866700 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:34.880817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.880762 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9973d9\"" Nov 03 20:15:34.891464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.891442 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9973d9-dockercfg-6cszh\"" Nov 03 20:15:34.895695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.895667 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn"] Nov 03 20:15:34.897022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.896997 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:34.897185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.897162 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.897252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.897211 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4mp\" (UniqueName: \"kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp\") pod \"payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:34.897307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.897251 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.897307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.897285 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2d5\" (UniqueName: \"kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.897408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.897382 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.897839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.897816 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.899817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.899794 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.900004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.899987 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.918039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.918018 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2d5\" (UniqueName: \"kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5\") pod \"maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:34.923147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.923121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:34.958239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.958211 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg"] Nov 03 20:15:34.961230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.961212 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:34.989222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:34.989147 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg"] Nov 03 20:15:35.026554 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.026503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:35.027860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.027809 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwlw\" (UniqueName: \"kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw\") pod \"payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.028008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.027887 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.028008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.027998 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:35.028136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.028125 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4mp\" (UniqueName: \"kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp\") pod \"payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:35.029398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.029367 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:35.046249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.046222 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4mp\" (UniqueName: \"kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp\") pod \"payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:35.109417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.109392 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:35.129619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.129598 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwlw\" (UniqueName: \"kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw\") pod \"payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.129715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.129632 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.130587 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.130564 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.147784 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.147759 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwlw\" (UniqueName: \"kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw\") pod \"payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.184429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.184401 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:35.278702 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.278666 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:35.308065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.308036 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" event={"ID":"261cb760-7907-4d77-acf3-2f2cd927d303","Type":"ContainerStarted","Data":"e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1"} Nov 03 20:15:35.308202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.308142 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" event={"ID":"261cb760-7907-4d77-acf3-2f2cd927d303","Type":"ContainerStarted","Data":"cafccfdf8f40bb9f14a639bdc1faf17dc407e004a53cebd79808627d40fab342"} Nov 03 20:15:35.308202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.308176 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:35.309955 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.309921 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" event={"ID":"0cdb90f0-5685-43db-892b-d7f803178ad1","Type":"ContainerStarted","Data":"3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9"} Nov 03 20:15:35.310107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.309972 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" event={"ID":"0cdb90f0-5685-43db-892b-d7f803178ad1","Type":"ContainerStarted","Data":"97262d210418faa544750598fee6b917542b0502d9c8bb0a1408273475e80732"} Nov 03 20:15:35.310460 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.310438 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:35.310520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.310489 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:35.389681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.389593 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" podStartSLOduration=2.389578814 podStartE2EDuration="2.389578814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:35.351920896 +0000 UTC m=+716.818876176" watchObservedRunningTime="2026-11-03 20:15:35.389578814 +0000 UTC m=+716.856534072" Nov 03 20:15:35.389946 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.389926 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" podStartSLOduration=2.389919216 podStartE2EDuration="2.389919216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:35.388631811 +0000 UTC m=+716.855587064" watchObservedRunningTime="2026-11-03 20:15:35.389919216 +0000 UTC m=+716.856874469" Nov 03 20:15:35.407519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.407490 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm"] Nov 03 20:15:35.411572 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:35.411550 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc6e0cd_9441_4d19_b553_19044c38ac0b.slice/crio-a4c8b6ed844abe44812066991776f646000302690fe0a245b50cb16f725fac33 WatchSource:0}: Error finding container a4c8b6ed844abe44812066991776f646000302690fe0a245b50cb16f725fac33: Status 404 returned error can't find the container with id a4c8b6ed844abe44812066991776f646000302690fe0a245b50cb16f725fac33 Nov 03 20:15:35.483848 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.483817 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn"] Nov 03 20:15:35.488569 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:35.488540 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eae762_603f_471d_8b1a_67fdb9fb6db8.slice/crio-e8f4c7d5bf3b3f2d6f72e845e8b614c262b703f59df96c51f89b90190fe5cdd0 WatchSource:0}: Error finding container e8f4c7d5bf3b3f2d6f72e845e8b614c262b703f59df96c51f89b90190fe5cdd0: Status 404 returned error can't find the container with id e8f4c7d5bf3b3f2d6f72e845e8b614c262b703f59df96c51f89b90190fe5cdd0 Nov 03 20:15:35.577711 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:35.577630 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg"] Nov 03 20:15:35.582926 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:35.582904 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53011f9e_2a42_4c14_a8e5_f8c45d12bf3c.slice/crio-23b24ec6a1aff7ea9458dc528b8d7b21138a0ea6e1a5379ca8ad2ede97ea3725 WatchSource:0}: Error finding container 23b24ec6a1aff7ea9458dc528b8d7b21138a0ea6e1a5379ca8ad2ede97ea3725: Status 404 returned error can't find the container with id 23b24ec6a1aff7ea9458dc528b8d7b21138a0ea6e1a5379ca8ad2ede97ea3725 Nov 03 20:15:35.630128 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:15:35.630098 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:15:36.242982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.242950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:36.318562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.318522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" event={"ID":"c6eae762-603f-471d-8b1a-67fdb9fb6db8","Type":"ContainerStarted","Data":"320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d"} Nov 03 20:15:36.318999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.318587 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" event={"ID":"c6eae762-603f-471d-8b1a-67fdb9fb6db8","Type":"ContainerStarted","Data":"e8f4c7d5bf3b3f2d6f72e845e8b614c262b703f59df96c51f89b90190fe5cdd0"} Nov 03 20:15:36.318999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.318640 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:36.320870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.320844 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" event={"ID":"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c","Type":"ContainerStarted","Data":"ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587"} Nov 03 20:15:36.320979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.320898 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:36.320979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.320920 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" event={"ID":"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c","Type":"ContainerStarted","Data":"23b24ec6a1aff7ea9458dc528b8d7b21138a0ea6e1a5379ca8ad2ede97ea3725"} Nov 03 20:15:36.322968 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.322945 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" event={"ID":"afc6e0cd-9441-4d19-b553-19044c38ac0b","Type":"ContainerStarted","Data":"c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75"} Nov 03 20:15:36.323052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.322989 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" event={"ID":"afc6e0cd-9441-4d19-b553-19044c38ac0b","Type":"ContainerStarted","Data":"a4c8b6ed844abe44812066991776f646000302690fe0a245b50cb16f725fac33"} Nov 03 20:15:36.323708 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.323690 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:36.341253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.341224 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:36.348436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.348413 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7"] Nov 03 20:15:36.351574 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.351557 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.359884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.359830 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" podStartSLOduration=2.3598143 podStartE2EDuration="2.3598143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:36.357225682 +0000 UTC m=+717.824180938" watchObservedRunningTime="2026-11-03 20:15:36.3598143 +0000 UTC m=+717.826769556" Nov 03 20:15:36.369079 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.369058 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e80203\"" Nov 03 20:15:36.371706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.371675 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7"] Nov 03 20:15:36.394247 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.394196 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" podStartSLOduration=1.9948092069999999 podStartE2EDuration="2.394178954s" totalImagesPullingTime="399.369747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:36.393585353 +0000 UTC m=+717.860540641" watchObservedRunningTime="2026-11-03 20:15:36.394178954 +0000 UTC m=+717.861134210" Nov 03 20:15:36.430502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.430459 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" podStartSLOduration=2.430447234 podStartE2EDuration="2.430447234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:36.429584407 +0000 UTC m=+717.896539660" watchObservedRunningTime="2026-11-03 20:15:36.430447234 +0000 UTC m=+717.897402547" Nov 03 20:15:36.444681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.444650 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.444836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.444815 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnnq\" (UniqueName: \"kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq\") pod \"payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.520970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.520891 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc"] Nov 03 20:15:36.525130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.525107 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.543365 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.543340 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc"] Nov 03 20:15:36.543570 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.543401 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e80203-serving-cert\"" Nov 03 20:15:36.546472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.546441 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.546576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.546488 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnnq\" (UniqueName: \"kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq\") pod \"payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.547765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.547739 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.573627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.573598 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnnq\" (UniqueName: \"kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq\") pod \"payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.615934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.615901 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:36.647128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.647087 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.647271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.647180 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.647271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.647252 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.647468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.647440 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgv59\" (UniqueName: \"kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.665514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.665493 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:36.716657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.716595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:36.749107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.748773 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv59\" (UniqueName: \"kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.749107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.748840 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.749107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.748889 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.749107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.748967 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.749756 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.749727 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.751949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.751921 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.752281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.752263 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.767367 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.767341 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv59\" (UniqueName: \"kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59\") pod \"maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:36.811374 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.811296 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:36.838101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:36.838042 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:37.007056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.007026 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7"] Nov 03 20:15:37.010467 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:37.010443 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba0abe0_719e_4208_a2d6_790a5c0dec00.slice/crio-d05276a9cfb62fbbb7519374cd1e0cb97c73ece637b6713aeaa43a9a60b07dda WatchSource:0}: Error finding container d05276a9cfb62fbbb7519374cd1e0cb97c73ece637b6713aeaa43a9a60b07dda: Status 404 returned error can't find the container with id d05276a9cfb62fbbb7519374cd1e0cb97c73ece637b6713aeaa43a9a60b07dda Nov 03 20:15:37.163453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.163409 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc"] Nov 03 20:15:37.173433 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:37.173402 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f72b17_0904_4497_8705_32c31f8b3b4d.slice/crio-b05958a90c625d363b4c2cf97e6f02f7e4b5aaa0623d7bd2dcc0fcfaeced881c WatchSource:0}: Error finding container b05958a90c625d363b4c2cf97e6f02f7e4b5aaa0623d7bd2dcc0fcfaeced881c: Status 404 returned error can't find the container with id b05958a90c625d363b4c2cf97e6f02f7e4b5aaa0623d7bd2dcc0fcfaeced881c Nov 03 20:15:37.335126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.335064 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" event={"ID":"0ba0abe0-719e-4208-a2d6-790a5c0dec00","Type":"ContainerStarted","Data":"615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd"} Nov 03 20:15:37.335551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.335267 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:37.335551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.335308 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" event={"ID":"0ba0abe0-719e-4208-a2d6-790a5c0dec00","Type":"ContainerStarted","Data":"d05276a9cfb62fbbb7519374cd1e0cb97c73ece637b6713aeaa43a9a60b07dda"} Nov 03 20:15:37.336404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.336374 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" event={"ID":"b5f72b17-0904-4497-8705-32c31f8b3b4d","Type":"ContainerStarted","Data":"b05958a90c625d363b4c2cf97e6f02f7e4b5aaa0623d7bd2dcc0fcfaeced881c"} Nov 03 20:15:37.371157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.371111 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" podStartSLOduration=1.371100001 podStartE2EDuration="1.371100001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:37.370122177 +0000 UTC m=+718.837077431" watchObservedRunningTime="2026-11-03 20:15:37.371100001 +0000 UTC m=+718.838055249" Nov 03 20:15:37.881208 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.881106 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:37.991747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:37.991711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.344884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.344850 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" event={"ID":"b5f72b17-0904-4497-8705-32c31f8b3b4d","Type":"ContainerStarted","Data":"b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862"} Nov 03 20:15:38.345377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.344940 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:38.383556 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.383507 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" podStartSLOduration=1.9858960780000001 podStartE2EDuration="2.383492949s" totalImagesPullingTime="397.596871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:38.382534242 +0000 UTC m=+719.849489497" watchObservedRunningTime="2026-11-03 20:15:38.383492949 +0000 UTC m=+719.850448203" Nov 03 20:15:38.428427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.428398 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb"] Nov 03 20:15:38.431541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.431520 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.445404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.445372 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-6fc748-serving-cert\"" Nov 03 20:15:38.452231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.452206 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb"] Nov 03 20:15:38.511699 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.511663 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.584248 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.584214 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.584394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.584354 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sd7x\" (UniqueName: \"kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.584452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.584419 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.584452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.584441 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.620406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.620323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.685652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.685622 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.685823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.685662 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.685823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.685721 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.685823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.685807 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sd7x\" (UniqueName: \"kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.686471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.686451 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.687985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.687960 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.700507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.700488 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-6fc748-serving-cert\"" Nov 03 20:15:38.707246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.707220 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sd7x\" (UniqueName: \"kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.708398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.708371 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.744831 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.744804 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:38.745629 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.745601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.859382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.859354 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.976710 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:38.976624 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:39.084495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:39.084462 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb"] Nov 03 20:15:39.088568 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:39.088541 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7574c52b_4d0d_477a_b151_0d24fd275e46.slice/crio-8fb948cfc4effd9af70c1cb64160f9b17fcfe8f91becfcee82de3a182f9a4b01 WatchSource:0}: Error finding container 8fb948cfc4effd9af70c1cb64160f9b17fcfe8f91becfcee82de3a182f9a4b01: Status 404 returned error can't find the container with id 8fb948cfc4effd9af70c1cb64160f9b17fcfe8f91becfcee82de3a182f9a4b01 Nov 03 20:15:39.354353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:39.354318 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" event={"ID":"7574c52b-4d0d-477a-b151-0d24fd275e46","Type":"ContainerStarted","Data":"8fb948cfc4effd9af70c1cb64160f9b17fcfe8f91becfcee82de3a182f9a4b01"} Nov 03 20:15:40.201926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.201889 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t"] Nov 03 20:15:40.205207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.205186 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.223261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.223238 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e40816-serving-cert\"" Nov 03 20:15:40.229159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.229127 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t"] Nov 03 20:15:40.284906 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.284865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:40.306051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.306019 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.306241 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.306062 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.306241 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.306223 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.306393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.306372 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttggp\" (UniqueName: \"kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.316307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.316284 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:15:40.367034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.367000 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" event={"ID":"7574c52b-4d0d-477a-b151-0d24fd275e46","Type":"ContainerStarted","Data":"a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334"} Nov 03 20:15:40.367487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.367105 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:40.387128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.387062 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:40.407490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.407462 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttggp\" (UniqueName: \"kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.407643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.407588 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.407712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.407641 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.407774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.407706 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.409551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.409497 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" podStartSLOduration=2.104771164 podStartE2EDuration="2.409481775s" totalImagesPullingTime="304.710611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.405705691 +0000 UTC m=+721.872660945" watchObservedRunningTime="2026-11-03 20:15:40.409481775 +0000 UTC m=+721.876437035" Nov 03 20:15:40.409656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.409549 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.411524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.411495 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.411875 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.411856 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.426300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.426279 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttggp\" (UniqueName: \"kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp\") pod \"maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.493730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.493665 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:40.517686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.517665 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:40.831607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:40.831442 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t"] Nov 03 20:15:40.834561 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:40.834534 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dfcaed_031e_4192_b2fb_e3a7a7cab90d.slice/crio-285c66b1c04810543bec534bb7c2ceb6ba0dad9db4e2362bc38793ec165cba82 WatchSource:0}: Error finding container 285c66b1c04810543bec534bb7c2ceb6ba0dad9db4e2362bc38793ec165cba82: Status 404 returned error can't find the container with id 285c66b1c04810543bec534bb7c2ceb6ba0dad9db4e2362bc38793ec165cba82 Nov 03 20:15:41.375246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:41.375206 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" event={"ID":"51dfcaed-031e-4192-b2fb-e3a7a7cab90d","Type":"ContainerStarted","Data":"031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c"} Nov 03 20:15:41.375633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:41.375272 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" event={"ID":"51dfcaed-031e-4192-b2fb-e3a7a7cab90d","Type":"ContainerStarted","Data":"285c66b1c04810543bec534bb7c2ceb6ba0dad9db4e2362bc38793ec165cba82"} Nov 03 20:15:41.375633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:41.375304 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:41.429805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:41.429699 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" podStartSLOduration=1.121662328 podStartE2EDuration="1.429678854s" totalImagesPullingTime="308.016526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:41.425742166 +0000 UTC m=+722.892697421" watchObservedRunningTime="2026-11-03 20:15:41.429678854 +0000 UTC m=+722.896634109" Nov 03 20:15:42.064499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.064461 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft"] Nov 03 20:15:42.068007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.067987 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.081984 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.081962 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-fd01ad-serving-cert\"" Nov 03 20:15:42.113223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.113192 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft"] Nov 03 20:15:42.141731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.141694 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:42.233634 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.233605 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.233759 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.233641 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574xk\" (UniqueName: \"kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.233859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.233834 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.233895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.233878 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.241330 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.241302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:42.335812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.335775 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.336008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.335836 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.336008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.335970 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.336162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.336009 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574xk\" (UniqueName: \"kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.336577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.336548 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.338491 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.338462 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.338619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.338523 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.354028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.354004 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574xk\" (UniqueName: \"kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk\") pod \"maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.369767 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.369737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:42.386280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.386257 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:42.693274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:42.693222 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft"] Nov 03 20:15:42.695787 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:42.695762 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25078489_ee49_4746_b11b_b203c00e6787.slice/crio-37bbea8078a524724f859fd993129ea43c5260398991ca69e372fbdfac669ef3 WatchSource:0}: Error finding container 37bbea8078a524724f859fd993129ea43c5260398991ca69e372fbdfac669ef3: Status 404 returned error can't find the container with id 37bbea8078a524724f859fd993129ea43c5260398991ca69e372fbdfac669ef3 Nov 03 20:15:43.291317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.291234 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:15:43.352084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.352037 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:15:43.389338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.389306 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" event={"ID":"25078489-ee49-4746-b11b-b203c00e6787","Type":"ContainerStarted","Data":"21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c"} Nov 03 20:15:43.389709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.389355 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" event={"ID":"25078489-ee49-4746-b11b-b203c00e6787","Type":"ContainerStarted","Data":"37bbea8078a524724f859fd993129ea43c5260398991ca69e372fbdfac669ef3"} Nov 03 20:15:43.389709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.389375 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:43.431571 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.431511 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" podStartSLOduration=1.140574507 podStartE2EDuration="1.431492417s" totalImagesPullingTime="290.91791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:43.426996952 +0000 UTC m=+724.893952254" watchObservedRunningTime="2026-11-03 20:15:43.431492417 +0000 UTC m=+724.898447672" Nov 03 20:15:43.921463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:43.921427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:44.302632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:44.302555 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:15:44.361688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:44.361661 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:15:45.314502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:45.314459 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:15:45.595739 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:45.595700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:45.695837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:45.695800 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:45.796728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:45.796683 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:46.326211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:46.326177 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:15:46.326639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:46.326322 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:15:46.381899 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:46.381872 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:15:47.339797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:47.339766 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:15:47.340240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:47.339947 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:15:47.340240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:47.339986 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:15:47.387783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:47.387753 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:15:47.435621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:47.435571 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:48.348965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:48.348937 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:15:49.235608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:49.235571 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:49.401971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:49.401938 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:15:51.220021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:51.219983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:53.340670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:53.340609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:54.529958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:54.529923 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:54.633791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:54.633753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.089747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.089644 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.223254 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.223216 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.363212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.363117 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.665563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.665484 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk"] Nov 03 20:15:55.669643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.669617 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.704058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.701705 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk"] Nov 03 20:15:55.765660 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.765612 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6fc748-openshift-default-dockercfg-h5hq6\"" Nov 03 20:15:55.786855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.786821 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqx9j\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.786923 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.786955 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.786981 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.787027 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.787066 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.787500 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.787539 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.787707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.787684 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.888836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.888798 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx9j\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.888845 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.888869 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.888885 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.888922 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.888961 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889045 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889098 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889227 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889393 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889448 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889594 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.889835 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.889659 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.890379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.890355 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.891898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.891875 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.891990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.891977 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.908190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.908165 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqx9j\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.908602 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.908573 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:55.985250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:55.985163 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:56.328429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.328393 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk"] Nov 03 20:15:56.333653 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:15:56.333625 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d463b1_7917_4095_9755_ee347e7e2955.slice/crio-cc3f8b5927f29ab460616eb7ec329c642b2d71035a375eec2b0f56a6f4542601 WatchSource:0}: Error finding container cc3f8b5927f29ab460616eb7ec329c642b2d71035a375eec2b0f56a6f4542601: Status 404 returned error can't find the container with id cc3f8b5927f29ab460616eb7ec329c642b2d71035a375eec2b0f56a6f4542601 Nov 03 20:15:56.335870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.335843 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:56.335962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.335905 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:56.335962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.335934 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:56.474553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.474519 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" event={"ID":"c0d463b1-7917-4095-9755-ee347e7e2955","Type":"ContainerStarted","Data":"900863454ad51169450c0db0ea5111576a0d32ab005f4191385a196990c0680f"} Nov 03 20:15:56.474730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.474574 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" event={"ID":"c0d463b1-7917-4095-9755-ee347e7e2955","Type":"ContainerStarted","Data":"cc3f8b5927f29ab460616eb7ec329c642b2d71035a375eec2b0f56a6f4542601"} Nov 03 20:15:56.515272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.515208 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podStartSLOduration=1.5151867 podStartE2EDuration="1.5151867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:56.513961247 +0000 UTC m=+737.980916512" watchObservedRunningTime="2026-11-03 20:15:56.5151867 +0000 UTC m=+737.982141957" Nov 03 20:15:56.713462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.713369 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:56.988159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:56.986205 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:15:57.210230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:57.210188 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:57.820825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:57.820786 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:57.991148 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:57.991111 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:15:57.991441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:57.991413 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:15:58.443106 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:58.439455 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:58.998578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:58.998546 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:15:58.999167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:58.999134 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:15:59.133009 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:59.132972 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:59.562696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:59.562598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:59.662136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:59.662097 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:59.765366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:59.765330 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:59.991096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:59.990413 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:15:59.991096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:15:59.990471 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:16:00.241853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:00.241780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:00.398014 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:00.397941 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:00.523165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:00.522964 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:00.661645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:00.661612 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:00.990464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:00.990423 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:16:00.990765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:00.990735 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:16:01.320950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:01.320869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:01.996096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:01.986415 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:16:01.996096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:01.986482 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:16:02.954980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:02.954936 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:02.991414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:02.991389 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:16:02.991782 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:02.991758 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:16:02.992509 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:02.992488 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:16:04.726199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:04.726158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:05.628168 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:16:05.628129 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:16:06.432761 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:06.432721 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:08.212321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:08.212281 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.125479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.125441 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.403770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.403680 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg"] Nov 03 20:16:10.407852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.407827 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.440103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.440045 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg"] Nov 03 20:16:10.464521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464487 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.464653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464554 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.464708 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464680 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.464804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464786 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnx6\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.464874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464856 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.464926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464913 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.464975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.464948 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.465030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.465001 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.465150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.465128 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.507875 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.507844 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-9e1bcb-openshift-default-dockercfg-sjgmc\"" Nov 03 20:16:10.566406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566378 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566434 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566473 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566606 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnx6\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566657 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566734 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566810 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566743 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566810 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566803 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.566996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.566971 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.567048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.567015 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.567136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.567043 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.567335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.567314 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.567412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.567372 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.567773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.567750 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.569025 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.569002 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.569317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.569294 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.587274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.587243 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.587519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.587498 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnx6\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:10.726176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:10.726093 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:11.054062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.053988 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg"] Nov 03 20:16:11.057426 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:16:11.057399 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6503bafd_9083_451e_b3a2_d8245349fbe1.slice/crio-e03047881b551f053822630c13ab84372966686b8ba40d47bff12dfde3ff30ed WatchSource:0}: Error finding container e03047881b551f053822630c13ab84372966686b8ba40d47bff12dfde3ff30ed: Status 404 returned error can't find the container with id e03047881b551f053822630c13ab84372966686b8ba40d47bff12dfde3ff30ed Nov 03 20:16:11.059797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.059765 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:11.059877 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.059835 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:11.059877 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.059864 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:11.617481 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.617442 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" event={"ID":"6503bafd-9083-451e-b3a2-d8245349fbe1","Type":"ContainerStarted","Data":"dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c"} Nov 03 20:16:11.617904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.617511 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" event={"ID":"6503bafd-9083-451e-b3a2-d8245349fbe1","Type":"ContainerStarted","Data":"e03047881b551f053822630c13ab84372966686b8ba40d47bff12dfde3ff30ed"} Nov 03 20:16:11.698007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.697935 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podStartSLOduration=1.697911826 podStartE2EDuration="1.697911826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:11.668757034 +0000 UTC m=+753.135712288" watchObservedRunningTime="2026-11-03 20:16:11.697911826 +0000 UTC m=+753.164867081" Nov 03 20:16:11.727007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.726967 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:11.849200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:11.849163 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:12.728849 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:12.728815 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:16:12.729385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:12.728894 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:16:13.652887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:13.652853 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:13.727400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:13.727370 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:16:13.727576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:13.727423 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:16:14.726951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:14.726908 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:16:14.727368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:14.726986 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:16:15.142173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:15.142138 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.444707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:15.444610 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.726940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:15.726854 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:16:15.726940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:15.726914 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:16:16.727236 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:16.727202 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:16:16.727651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:16.727258 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 20:16:17.078461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:17.078379 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:17.250569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:17.250526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:17.731349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:17.731322 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:17.731742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:17.731724 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:17.732350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:17.732328 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:16:19.185779 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:19.185745 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:19.539995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:19.539897 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:20.213867 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:20.213828 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:20.744536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:20.744495 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:20.858288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:20.858258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:20.962176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:20.962096 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.331869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:22.331834 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.659022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:22.658938 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:24.527421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:24.527379 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:25.763677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.763646 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv"] Nov 03 20:16:25.768728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.768703 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.795824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.795788 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv"] Nov 03 20:16:25.866769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.866738 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.866935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.866780 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.866935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.866816 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.867022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.866995 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.867161 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.867142 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.867646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.867615 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.867788 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.867680 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lr5\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.867860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.867791 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.867922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.867871 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.969516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.969473 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.969835 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.969811 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.969977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.969950 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.969972 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970297 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970316 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970498 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970747 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970785 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lr5\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.970924 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970835 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.971447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.971063 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.971447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.971223 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.971447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.971272 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.971692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.970509 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.972764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.972739 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.973790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.973766 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.990176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.990135 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lr5\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:25.991243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:25.991220 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:26.138356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.138324 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:26.256659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.256619 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:26.453304 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.453219 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv"] Nov 03 20:16:26.464470 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:16:26.464436 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8775583_9b9d_4f0c_a3b2_66584cf64490.slice/crio-a33586cdcdbfaaa4567bf20ad05ed20ea92f578ad356ed0b6d9771b6408df242 WatchSource:0}: Error finding container a33586cdcdbfaaa4567bf20ad05ed20ea92f578ad356ed0b6d9771b6408df242: Status 404 returned error can't find the container with id a33586cdcdbfaaa4567bf20ad05ed20ea92f578ad356ed0b6d9771b6408df242 Nov 03 20:16:26.466932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.466901 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:26.467029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.466967 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:26.467029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.466994 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:26.716204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.716117 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" event={"ID":"d8775583-9b9d-4f0c-a3b2-66584cf64490","Type":"ContainerStarted","Data":"ba0288d22592b67422da942aa2e98e1cd1112b85251baf07397a2ff56bc759e2"} Nov 03 20:16:26.716204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.716184 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" event={"ID":"d8775583-9b9d-4f0c-a3b2-66584cf64490","Type":"ContainerStarted","Data":"a33586cdcdbfaaa4567bf20ad05ed20ea92f578ad356ed0b6d9771b6408df242"} Nov 03 20:16:26.756047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:26.755993 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podStartSLOduration=1.755978324 podStartE2EDuration="1.755978324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:26.754802509 +0000 UTC m=+768.221757773" watchObservedRunningTime="2026-11-03 20:16:26.755978324 +0000 UTC m=+768.222933579" Nov 03 20:16:27.138564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:27.138528 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:27.140390 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:27.140358 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:27.140536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:27.140415 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:16:28.067744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:28.067691 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:28.139246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:28.139208 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:28.139709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:28.139276 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:16:29.139635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:29.139598 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:29.140060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:29.139656 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:16:29.649526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:29.649486 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:29.994561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:29.994480 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:30.138855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:30.138823 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:30.139020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:30.138875 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:16:31.139437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:31.139403 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:31.139840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:31.139457 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:16:31.647546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:31.647507 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:32.138894 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:32.138835 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:32.139120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:32.138900 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 20:16:33.143618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:33.143588 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:33.144123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:33.144104 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:33.144744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:33.144728 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:16:33.376351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:33.376309 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:34.269865 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:34.269829 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:34.990176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:34.990089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:35.628865 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:16:35.628832 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:16:36.707557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:36.707524 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:38.406546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:38.406500 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:39.881896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:39.881860 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:39.981713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:39.981677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:40.853901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.853862 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp"] Nov 03 20:16:40.860156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.860134 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.880035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.880014 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp"] Nov 03 20:16:40.968808 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.968780 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.968821 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dzv\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.968939 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.968972 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.969055 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.969098 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.969158 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.969246 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:40.969378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:40.969275 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.070690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070654 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.070878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070710 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.070878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070769 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.070878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070806 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.070878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070855 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070911 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.070947 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.071010 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.071052 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dzv\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.071603 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.071818 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.071895 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.071927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.071896 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.072231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.072186 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.073302 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.073274 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.073504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.073484 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.091221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.091194 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.092732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.092705 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dzv\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.173882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.173811 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:41.495616 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.495527 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp"] Nov 03 20:16:41.499921 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:16:41.499894 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a315a8_1afb_496f_a723_81beae6b1d38.slice/crio-c20bd6bfc5dbf8d707e5230c49cfee28502eb152f61662b4155f7165676f9a99 WatchSource:0}: Error finding container c20bd6bfc5dbf8d707e5230c49cfee28502eb152f61662b4155f7165676f9a99: Status 404 returned error can't find the container with id c20bd6bfc5dbf8d707e5230c49cfee28502eb152f61662b4155f7165676f9a99 Nov 03 20:16:41.503394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.503360 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:41.503498 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.503426 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:41.503498 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.503457 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:41.823578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.823477 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" event={"ID":"70a315a8-1afb-496f-a723-81beae6b1d38","Type":"ContainerStarted","Data":"f6f242bd8673b77927cf1d924bb4f0bfc6e40547e4ea10aef79fc690ca6add05"} Nov 03 20:16:41.823578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.823544 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" event={"ID":"70a315a8-1afb-496f-a723-81beae6b1d38","Type":"ContainerStarted","Data":"c20bd6bfc5dbf8d707e5230c49cfee28502eb152f61662b4155f7165676f9a99"} Nov 03 20:16:41.842324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.842295 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:41.866997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:41.866942 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podStartSLOduration=1.8669231659999999 podStartE2EDuration="1.866923166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:41.865537236 +0000 UTC m=+783.332492486" watchObservedRunningTime="2026-11-03 20:16:41.866923166 +0000 UTC m=+783.333878420" Nov 03 20:16:42.174854 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:42.174767 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:42.176205 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:42.176179 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:42.176315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:42.176236 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:16:43.174339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:43.174300 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:43.174510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:43.174365 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:16:43.558334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:43.558217 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.803665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:43.803626 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:44.174438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:44.174405 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:44.174635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:44.174463 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:16:45.174490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:45.174453 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:45.174881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:45.174515 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:16:45.573709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:45.573680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:46.175006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:46.174970 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:46.175442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:46.175031 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:16:47.174591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:47.174558 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:47.174759 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:47.174608 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:16:47.401771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:47.401737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:47.495356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:47.495267 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:48.178947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:48.178915 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:48.179433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:48.179411 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:48.180114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:48.180092 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:16:49.025154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:49.025115 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:50.839607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:50.839573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:50.940929 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:50.940896 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:52.621737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:52.621680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:54.228823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:54.228780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:55.581530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.581494 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm"] Nov 03 20:16:55.593399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.593365 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.624420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.624389 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm"] Nov 03 20:16:55.667888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.667845 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.667888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.667887 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668005 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668060 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668164 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668231 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zhz\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668248 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668267 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.668473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.668448 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.769913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.769879 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.769913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.769915 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.769963 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.769998 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770136 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770278 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zhz\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770308 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770343 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770395 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770405 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770426 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770485 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.770819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.770757 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.771041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.771016 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.772621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.772594 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.772773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.772751 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.788502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.788479 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.788633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.788581 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zhz\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:55.851244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.851155 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:55.913653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:55.912374 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:56.232837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:56.232750 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm"] Nov 03 20:16:56.239144 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:16:56.239041 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0c0970_75c7_4ae1_a683_8d85ebc90779.slice/crio-b6378f5843d61dc1527d4f0c50cb5ab6f0e6499bdeb71cf656a65f82f381eb8b WatchSource:0}: Error finding container b6378f5843d61dc1527d4f0c50cb5ab6f0e6499bdeb71cf656a65f82f381eb8b: Status 404 returned error can't find the container with id b6378f5843d61dc1527d4f0c50cb5ab6f0e6499bdeb71cf656a65f82f381eb8b Nov 03 20:16:56.241846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:56.241806 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:56.241941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:56.241903 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:56.241980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:56.241949 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:56.939674 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:56.939632 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" event={"ID":"0f0c0970-75c7-4ae1-a683-8d85ebc90779","Type":"ContainerStarted","Data":"e8c179a1cf4560d99b3cb0a2e364e79ff80d7128a1ad04c5bcfd3a434428b77a"} Nov 03 20:16:56.940207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:56.939692 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" event={"ID":"0f0c0970-75c7-4ae1-a683-8d85ebc90779","Type":"ContainerStarted","Data":"b6378f5843d61dc1527d4f0c50cb5ab6f0e6499bdeb71cf656a65f82f381eb8b"} Nov 03 20:16:57.007704 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:57.007641 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podStartSLOduration=2.007621404 podStartE2EDuration="2.007621404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:56.993625887 +0000 UTC m=+798.460581186" watchObservedRunningTime="2026-11-03 20:16:57.007621404 +0000 UTC m=+798.474576660" Nov 03 20:16:57.533979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:57.533944 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:57.913391 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:57.913350 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:16:57.915431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:57.915392 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:16:57.915576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:57.915459 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:16:58.913966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:58.913927 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:16:58.914507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:58.913994 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:16:59.238400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:59.238290 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:59.913886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:59.913851 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:16:59.914060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:16:59.913916 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:17:00.913346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:00.913304 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:17:00.913559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:00.913361 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:17:00.936022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:00.935991 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.612179 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:01.608106 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.913777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:01.913676 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:17:01.913777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:01.913747 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:17:02.714657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:02.714622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:02.917934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:02.917904 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:17:02.918313 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:02.918291 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:17:02.919053 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:02.919032 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:17:04.530121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:04.530057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:05.629049 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:17:05.628997 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:17:06.125305 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:06.125269 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:07.709358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:07.709309 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:09.203639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:09.203600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:10.829387 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:10.829354 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:12.433024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:12.432985 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:14.012226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:14.012175 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:15.713302 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:15.713268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:16.406466 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:16.406429 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:17.326367 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:17.326331 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:18.859607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:18.859572 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:20.507428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:20.507394 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:22.024144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:22.024109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.622398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:23.622358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:25.534116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:25.534066 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.113552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.113516 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq"] Nov 03 20:17:26.117943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.117914 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.150277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.150246 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e40816-openshift-default-dockercfg-rbqqp\"" Nov 03 20:17:26.151634 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.151608 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq"] Nov 03 20:17:26.161494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161451 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.161666 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161629 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.161764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161663 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.161764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161698 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.161910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161822 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsrcp\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.161910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161862 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.161910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161904 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.162212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.161963 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.162212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.162009 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263370 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263340 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263373 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263397 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263445 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsrcp\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263464 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263491 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.263560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263532 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263574 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263650 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263820 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.263883 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.264014 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.264037 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.264404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.264331 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.265704 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.265686 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.265992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.265973 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.282701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.282681 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsrcp\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.285435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.285414 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.433651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.433568 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:26.791633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.791589 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq"] Nov 03 20:17:26.794398 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:17:26.794369 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753bcdc7_60a4_4d92_9ca4_e11a33d4e339.slice/crio-bb4122ed9ddb178891bd2ac860b7e4c2d1774c2c45b8c7686ed4033eed5bca11 WatchSource:0}: Error finding container bb4122ed9ddb178891bd2ac860b7e4c2d1774c2c45b8c7686ed4033eed5bca11: Status 404 returned error can't find the container with id bb4122ed9ddb178891bd2ac860b7e4c2d1774c2c45b8c7686ed4033eed5bca11 Nov 03 20:17:26.796675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.796646 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:26.796730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.796708 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:26.796770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:26.796737 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:27.182997 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.182949 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" event={"ID":"753bcdc7-60a4-4d92-9ca4-e11a33d4e339","Type":"ContainerStarted","Data":"7aa33a2dcf23cf73fb2bb2667c85d2c98a4f96d32b0b90d29546da6f3d5c0e47"} Nov 03 20:17:27.183201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.183024 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" event={"ID":"753bcdc7-60a4-4d92-9ca4-e11a33d4e339","Type":"ContainerStarted","Data":"bb4122ed9ddb178891bd2ac860b7e4c2d1774c2c45b8c7686ed4033eed5bca11"} Nov 03 20:17:27.223250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.223212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:27.226485 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.226379 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podStartSLOduration=1.226366894 podStartE2EDuration="1.226366894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:27.223385081 +0000 UTC m=+828.690340353" watchObservedRunningTime="2026-11-03 20:17:27.226366894 +0000 UTC m=+828.693322148" Nov 03 20:17:27.436523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.433980 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:27.436791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.436759 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:17:27.436905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:27.436821 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:17:28.434712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:28.434675 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:17:28.435234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:28.434731 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:17:28.953476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:28.953433 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:29.434597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:29.434561 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:17:29.434807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:29.434628 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:17:29.765960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:29.765880 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:30.434647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:30.434612 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:17:30.434831 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:30.434675 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:17:30.762319 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:30.762225 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.433957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:31.433919 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:17:31.434169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:31.433979 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:17:32.434357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:32.434325 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:17:32.434739 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:32.434377 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 20:17:32.445460 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:32.445429 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:33.638644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:33.638598 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:33.639818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:33.639787 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:33.640914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:33.640887 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:17:34.086950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:34.086909 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:35.631538 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:17:35.631504 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:17:35.716876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:35.716832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:37.206307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:37.206272 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:38.903936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:38.903895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:40.493239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.493204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:40.687765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.687731 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8"] Nov 03 20:17:40.694330 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.694306 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.717436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.717410 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8"] Nov 03 20:17:40.786543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786453 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786502 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786599 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786640 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrvv\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786672 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786794 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786870 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786853 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.786941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.786924 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.787044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.787023 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.887988 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrvv\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888025 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888045 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888094 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888149 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888206 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888326 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888367 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888581 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888548 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888576 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888603 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888548 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.888747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.888649 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.889105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.889088 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.890472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.890451 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.890629 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.890613 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.907208 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.907182 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:40.907338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:40.907224 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrvv\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:41.008619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.008588 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:41.333959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.333927 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59"] Nov 03 20:17:41.340123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.340100 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8"] Nov 03 20:17:41.340252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.340154 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.351005 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:17:41.350978 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076e74c9_625f_40d8_a4a3_0488b0f6c228.slice/crio-60fddda94bb3e74140410404ad8cdab8a24a647a21d017a155e1cd7c928ab262 WatchSource:0}: Error finding container 60fddda94bb3e74140410404ad8cdab8a24a647a21d017a155e1cd7c928ab262: Status 404 returned error can't find the container with id 60fddda94bb3e74140410404ad8cdab8a24a647a21d017a155e1cd7c928ab262 Nov 03 20:17:41.354405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.354366 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:41.354510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.354447 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:41.354510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.354482 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:41.356857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.356832 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e80203-openshift-default-dockercfg-6kpjl\"" Nov 03 20:17:41.363203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.361328 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59"] Nov 03 20:17:41.496993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.496955 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497004 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c556l\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497154 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497198 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497304 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497353 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497462 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497504 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.497898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.497540 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598306 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598432 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598476 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598512 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598586 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598611 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c556l\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598667 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598694 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.598983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.598798 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.599705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.599677 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.599939 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.599919 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.602523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.602435 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.603061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.602643 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.603061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.602786 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.603061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.603011 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.603061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.603028 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.618608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.618583 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.620257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.620235 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c556l\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:41.658378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:41.658348 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:42.011717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.010715 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59"] Nov 03 20:17:42.014115 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:17:42.013567 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae01033_875e_4913_9671_9a2464a837a6.slice/crio-6de22410a47032a3cd1e3b013af68782fd679fca838b1e2f675b0ce0bc3a43c1 WatchSource:0}: Error finding container 6de22410a47032a3cd1e3b013af68782fd679fca838b1e2f675b0ce0bc3a43c1: Status 404 returned error can't find the container with id 6de22410a47032a3cd1e3b013af68782fd679fca838b1e2f675b0ce0bc3a43c1 Nov 03 20:17:42.016621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.016589 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:42.016737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.016665 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:42.016737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.016708 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:42.325849 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.325796 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" event={"ID":"9ae01033-875e-4913-9671-9a2464a837a6","Type":"ContainerStarted","Data":"81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c"} Nov 03 20:17:42.326130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.325909 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" event={"ID":"9ae01033-875e-4913-9671-9a2464a837a6","Type":"ContainerStarted","Data":"6de22410a47032a3cd1e3b013af68782fd679fca838b1e2f675b0ce0bc3a43c1"} Nov 03 20:17:42.328180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.328147 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" event={"ID":"076e74c9-625f-40d8-a4a3-0488b0f6c228","Type":"ContainerStarted","Data":"c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0"} Nov 03 20:17:42.328285 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.328208 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" event={"ID":"076e74c9-625f-40d8-a4a3-0488b0f6c228","Type":"ContainerStarted","Data":"60fddda94bb3e74140410404ad8cdab8a24a647a21d017a155e1cd7c928ab262"} Nov 03 20:17:42.352573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.352537 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:42.370694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.370646 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podStartSLOduration=1.370632927 podStartE2EDuration="1.370632927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:42.368663938 +0000 UTC m=+843.835619202" watchObservedRunningTime="2026-11-03 20:17:42.370632927 +0000 UTC m=+843.837588180" Nov 03 20:17:42.414920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.414865 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podStartSLOduration=2.414850031 podStartE2EDuration="2.414850031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:42.412183168 +0000 UTC m=+843.879138439" watchObservedRunningTime="2026-11-03 20:17:42.414850031 +0000 UTC m=+843.881805351" Nov 03 20:17:42.659439 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:42.659366 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:43.019143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:43.018240 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:43.023093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:43.019689 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:17:43.023093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:43.019744 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:17:43.660667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:43.660327 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:17:43.660667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:43.660396 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:17:44.023155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:44.020271 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:17:44.023155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:44.020327 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:17:44.334438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:44.332440 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:44.659232 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:44.659149 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:17:44.659384 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:44.659230 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:17:45.018696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:45.018178 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:17:45.018696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:45.018241 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:17:45.659084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:45.659033 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:17:45.661040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:45.659130 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:17:46.018573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:46.018478 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:17:46.019260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:46.018846 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:17:46.240226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:46.240142 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:46.659606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:46.659559 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:17:46.659816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:46.659625 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:17:47.019150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:47.015524 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:17:47.019150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:47.015601 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:17:47.658859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:47.658817 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:17:47.659042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:47.658890 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:17:48.011522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:48.011428 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:17:48.011796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:48.011760 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 20:17:48.148691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:48.148656 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:48.659494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:48.659459 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:17:48.659701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:48.659517 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 20:17:49.013999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.013968 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:49.014375 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.014317 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:49.015200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.015178 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:17:49.662996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.662966 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:49.663468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.663265 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:49.663975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.663958 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:17:49.826975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:49.826928 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:51.751461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:51.751218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:53.616353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:53.616306 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:55.298543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:55.298508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:56.909636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:56.909596 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:58.613773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:17:58.613731 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:00.212609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:00.212577 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:01.710994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:01.710956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.335623 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:03.335588 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:05.016549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:05.016512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:05.630574 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:18:05.630538 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:18:06.863170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:06.863049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:09.348292 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:09.348248 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.619922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.619884 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz"] Nov 03 20:18:10.626201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.626171 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.640047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.640018 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fd01ad-openshift-default-dockercfg-lbghm\"" Nov 03 20:18:10.656430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.656388 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz"] Nov 03 20:18:10.675590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.675544 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.675764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.675593 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpzv\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.675764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.675727 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.675890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.675799 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.675960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.675891 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.676051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.676034 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.676186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.676168 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.676297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.676275 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.676451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.676313 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.776988 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.776949 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.776988 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.776989 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpzv\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777129 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777240 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777270 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777408 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777497 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777601 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777682 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777640 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777682 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777638 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777682 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777664 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.777950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.777922 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.778129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.778056 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.778250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.778221 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.780507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.780481 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.783121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.783094 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.796953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.796920 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.797144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.797013 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpzv\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:10.941261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:10.941179 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:11.235002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.234927 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.267239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.267203 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz"] Nov 03 20:18:11.271469 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:11.271438 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b66086_db00_4280_8a94_e515abf810ab.slice/crio-9f86ce2fe4a8e4df7dc4759a8d7846627b96c642b2bb23b94f5227b929f6a61c WatchSource:0}: Error finding container 9f86ce2fe4a8e4df7dc4759a8d7846627b96c642b2bb23b94f5227b929f6a61c: Status 404 returned error can't find the container with id 9f86ce2fe4a8e4df7dc4759a8d7846627b96c642b2bb23b94f5227b929f6a61c Nov 03 20:18:11.274368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.274335 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:11.274495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.274405 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:11.274495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.274437 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:11.307632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.307589 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp"] Nov 03 20:18:11.316419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.316388 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.335125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.332560 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp"] Nov 03 20:18:11.340166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.340138 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-482722-openshift-default-dockercfg-6d2xm\"" Nov 03 20:18:11.346545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.345869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.384996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.384924 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385013 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385085 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385151 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385202 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7nx\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385253 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385274 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385380 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.385448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.385416 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.486729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486637 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.486729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486680 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486726 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486810 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486906 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7nx\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486957 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.486983 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487016 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487042 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487295 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487393 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487490 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487499 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.487736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.487725 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.490062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.490039 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.490433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.490412 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.505536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.505508 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.505720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.505699 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7nx\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-dqmqp\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.664306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.664272 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:11.813619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.813515 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" event={"ID":"25b66086-db00-4280-8a94-e515abf810ab","Type":"ContainerStarted","Data":"f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01"} Nov 03 20:18:11.813774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.813644 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" event={"ID":"25b66086-db00-4280-8a94-e515abf810ab","Type":"ContainerStarted","Data":"9f86ce2fe4a8e4df7dc4759a8d7846627b96c642b2bb23b94f5227b929f6a61c"} Nov 03 20:18:11.896984 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.884032 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podStartSLOduration=1.884012437 podStartE2EDuration="1.884012437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:11.880214735 +0000 UTC m=+873.347170015" watchObservedRunningTime="2026-11-03 20:18:11.884012437 +0000 UTC m=+873.350967691" Nov 03 20:18:11.941513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:11.941480 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:12.012423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.012396 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp"] Nov 03 20:18:12.013777 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:12.013747 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4d35fe_8d90_429f_9713_315e2b947b80.slice/crio-21ae59ecafdebc637de4c9f5a1a689fd849f115b705d87007ba868f00ff24103 WatchSource:0}: Error finding container 21ae59ecafdebc637de4c9f5a1a689fd849f115b705d87007ba868f00ff24103: Status 404 returned error can't find the container with id 21ae59ecafdebc637de4c9f5a1a689fd849f115b705d87007ba868f00ff24103 Nov 03 20:18:12.016500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.016469 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:12.016576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.016535 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:12.016576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.016563 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:12.914095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.905979 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" event={"ID":"2f4d35fe-8d90-429f-9713-315e2b947b80","Type":"ContainerStarted","Data":"3cbfd9f537eac4ef8a0c329051273ba1845c779033179c6101f9ac0dfe2024ad"} Nov 03 20:18:12.914095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.906037 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" event={"ID":"2f4d35fe-8d90-429f-9713-315e2b947b80","Type":"ContainerStarted","Data":"21ae59ecafdebc637de4c9f5a1a689fd849f115b705d87007ba868f00ff24103"} Nov 03 20:18:12.946695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.946376 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:18:12.946695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:12.946436 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:18:13.007057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.006988 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podStartSLOduration=2.006964631 podStartE2EDuration="2.006964631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:12.978247282 +0000 UTC m=+874.445202536" watchObservedRunningTime="2026-11-03 20:18:13.006964631 +0000 UTC m=+874.473919887" Nov 03 20:18:13.223152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.219602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:13.665385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.665350 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:13.667920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.667889 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:18:13.668048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.667952 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 20:18:13.946144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.945448 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:18:13.946144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:13.945524 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:18:14.665669 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:14.665626 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:18:14.665862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:14.665689 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 20:18:14.941669 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:14.941591 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:18:14.941669 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:14.941648 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:18:15.054214 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:15.054172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:15.153334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:15.153292 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:15.665283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:15.665242 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:18:15.665535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:15.665506 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 20:18:15.942446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:15.942365 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:18:15.942645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:15.942621 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:18:16.665083 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:16.665024 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:18:16.665606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:16.665102 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 20:18:16.942062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:16.941978 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:18:16.942062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:16.942029 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:18:16.966055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:16.966028 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.664637 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:17.664602 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:18:17.664827 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:17.664659 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 20:18:17.942312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:17.942230 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:18:17.942312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:17.942285 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 20:18:18.664595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:18.664560 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:18:18.664758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:18.664607 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 20:18:18.766905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:18.766863 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:18.947334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:18.947235 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:18.947750 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:18.947599 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:18.948287 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:18.948262 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:18:19.668717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:19.668687 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:19.669013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:19.668994 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:19.669649 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:19.669623 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:18:20.404441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:20.404408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:22.376114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:22.375895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:25.843098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:25.842985 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:27.543391 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.543348 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:27.777876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.777841 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5"] Nov 03 20:18:27.783606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.783580 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.806043 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.806012 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5"] Nov 03 20:18:27.863999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.863969 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.863999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864001 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccc348-aabb-4a1d-a3d5-f8826969be95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864021 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjm8\" (UniqueName: \"kubernetes.io/projected/02ccc348-aabb-4a1d-a3d5-f8826969be95-kube-api-access-jnjm8\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864041 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864187 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864208 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864277 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864322 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.864476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.864463 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966175 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966318 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966343 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccc348-aabb-4a1d-a3d5-f8826969be95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966367 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjm8\" (UniqueName: \"kubernetes.io/projected/02ccc348-aabb-4a1d-a3d5-f8826969be95-kube-api-access-jnjm8\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966394 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966434 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966462 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966562 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.966852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966765 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.967021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966996 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.967185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.966996 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.967185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.967110 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.967344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.967325 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccc348-aabb-4a1d-a3d5-f8826969be95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.969294 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.969271 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.969632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.969604 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.985251 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.985225 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccc348-aabb-4a1d-a3d5-f8826969be95-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:27.985348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:27.985320 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjm8\" (UniqueName: \"kubernetes.io/projected/02ccc348-aabb-4a1d-a3d5-f8826969be95-kube-api-access-jnjm8\") pod \"maas-default-gateway-openshift-default-694db55c87-rmjj5\" (UID: \"02ccc348-aabb-4a1d-a3d5-f8826969be95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:28.101716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:28.101677 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:28.420331 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:28.420206 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5"] Nov 03 20:18:28.422869 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:28.422842 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ccc348_aabb_4a1d_a3d5_f8826969be95.slice/crio-a2fdb0fd2b4bcbc86e5e40aeeb83f0b778b81cb81e32f712df2fd25d1d907b62 WatchSource:0}: Error finding container a2fdb0fd2b4bcbc86e5e40aeeb83f0b778b81cb81e32f712df2fd25d1d907b62: Status 404 returned error can't find the container with id a2fdb0fd2b4bcbc86e5e40aeeb83f0b778b81cb81e32f712df2fd25d1d907b62 Nov 03 20:18:28.425466 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:28.425427 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:28.425579 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:28.425500 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:28.425579 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:28.425538 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:29.172413 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:29.172373 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:29.301096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:29.298629 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" event={"ID":"02ccc348-aabb-4a1d-a3d5-f8826969be95","Type":"ContainerStarted","Data":"608dc66f157fee7a4a535b2b9686be0ddd3ec4ad8abd37c571e6d407417f989a"} Nov 03 20:18:29.301096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:29.298697 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" event={"ID":"02ccc348-aabb-4a1d-a3d5-f8826969be95","Type":"ContainerStarted","Data":"a2fdb0fd2b4bcbc86e5e40aeeb83f0b778b81cb81e32f712df2fd25d1d907b62"} Nov 03 20:18:29.365428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:29.365374 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podStartSLOduration=2.365358656 podStartE2EDuration="2.365358656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:29.350873428 +0000 UTC m=+890.817828693" watchObservedRunningTime="2026-11-03 20:18:29.365358656 +0000 UTC m=+890.832313910" Nov 03 20:18:29.882121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:29.878332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:30.104244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:30.104201 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:30.106502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:30.106466 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:18:30.106636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:30.106531 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 20:18:30.847151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:30.847108 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.104218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:31.103500 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:18:31.104218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:31.103561 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 20:18:32.103502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:32.103458 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:18:32.103950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:32.103518 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 20:18:32.643976 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:32.643939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:33.104094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:33.103716 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:18:33.104094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:33.103815 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 20:18:34.104095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:34.103511 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:18:34.104095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:34.103579 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 20:18:34.353985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:34.353949 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:35.102151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:35.102111 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:18:35.102353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:35.102181 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 20:18:35.629641 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:18:35.629600 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:18:35.934673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:35.934579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:37.056656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:37.056619 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:37.057252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:37.057105 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:37.058290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:37.058268 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" Nov 03 20:18:37.522661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:37.522572 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:37.621579 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:37.621545 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:38.795220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:38.795194 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:18:38.800805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:38.800780 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:18:38.804150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:38.804128 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:18:38.810390 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:38.810371 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:18:39.123636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:39.123603 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:41.140099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.140046 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t"] Nov 03 20:18:41.190885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.190858 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t"] Nov 03 20:18:41.191233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.191212 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.231098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.230833 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9973d9-openshift-default-dockercfg-br8jj\"" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.366822 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4vn\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.366965 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367054 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367109 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367136 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367178 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367223 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367293 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.367350 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.426170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.407678 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469348 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469451 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469490 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469520 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469562 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469606 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469674 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469729 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.470110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.469782 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4vn\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.472244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.472051 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.472244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.472194 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.472446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.472365 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.478652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.476556 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.478872 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.476921 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.480170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.479297 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.490381 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.490354 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.524850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.524817 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4vn\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.542241 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.542204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:41.553434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.553392 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:41.686194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:41.685857 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:42.011573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.011534 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t"] Nov 03 20:18:42.014478 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:42.014449 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2859563_48c2_429d_ba23_b22f5b16b30e.slice/crio-e5ebb19daed9c9310874e487744603d7f0e66a830e3dc973ff0c0e1e519282ce WatchSource:0}: Error finding container e5ebb19daed9c9310874e487744603d7f0e66a830e3dc973ff0c0e1e519282ce: Status 404 returned error can't find the container with id e5ebb19daed9c9310874e487744603d7f0e66a830e3dc973ff0c0e1e519282ce Nov 03 20:18:42.017641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.017608 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:42.017746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.017677 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:42.017746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.017707 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:42.491107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.491048 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" event={"ID":"e2859563-48c2-429d-ba23-b22f5b16b30e","Type":"ContainerStarted","Data":"c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984"} Nov 03 20:18:42.491548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.491184 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" event={"ID":"e2859563-48c2-429d-ba23-b22f5b16b30e","Type":"ContainerStarted","Data":"e5ebb19daed9c9310874e487744603d7f0e66a830e3dc973ff0c0e1e519282ce"} Nov 03 20:18:42.546776 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.546714 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podStartSLOduration=1.546697776 podStartE2EDuration="1.546697776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:42.543120233 +0000 UTC m=+904.010075501" watchObservedRunningTime="2026-11-03 20:18:42.546697776 +0000 UTC m=+904.013653029" Nov 03 20:18:42.686283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:42.686253 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:43.120131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:43.120095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:43.695180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:43.695139 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:43.695781 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:43.695749 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:18:44.700098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:44.697446 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:44.700098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:44.697513 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:18:44.867901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:44.867868 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:45.688510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:45.688468 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:45.688812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:45.688779 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:18:46.534212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:46.534094 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:46.691095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:46.686441 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:46.691095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:46.686510 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:18:47.689096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:47.686383 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:47.689096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:47.686444 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:18:48.065013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.064933 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s"] Nov 03 20:18:48.071718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.071695 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.117612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.108125 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s"] Nov 03 20:18:48.117867 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.116426 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"distinct-w0-e40816-kserve-self-signed-certs\"" Nov 03 20:18:48.117987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.116531 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"openshift-service-ca.crt\"" Nov 03 20:18:48.118137 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.116979 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"default-dockercfg-2vgst\"" Nov 03 20:18:48.118211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.117092 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"kube-root-ca.crt\"" Nov 03 20:18:48.127908 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.127876 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:48.218663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.218620 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:48.233824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.233791 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.234016 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.233882 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.234121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.234095 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkzf\" (UniqueName: \"kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.234181 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.234149 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.234300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.234280 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.234368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.234314 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.335664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.335528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkzf\" (UniqueName: \"kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.335664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.335593 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.335664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.335694 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.335664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.335726 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.336486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.335853 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.336486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.335974 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.338986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.336615 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.338986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.336799 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.338986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.337340 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.343133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.340177 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.343133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.340412 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.358847 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.356865 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkzf\" (UniqueName: \"kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf\") pod \"distinct-w0-e40816-kserve-6b4c76b47-9qv4s\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.436439 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.436332 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:48.686862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.686787 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:48.686862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.686839 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:18:48.759146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:48.758650 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s"] Nov 03 20:18:49.564130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.564089 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" event={"ID":"3144a231-859a-4528-9c96-d9e60053adc8","Type":"ContainerStarted","Data":"acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e"} Nov 03 20:18:49.564320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.564142 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" event={"ID":"3144a231-859a-4528-9c96-d9e60053adc8","Type":"ContainerStarted","Data":"f34984cde3a23a6d1235cf2f6d1e3fb11658cc8479482f1d11bd75b13182ced6"} Nov 03 20:18:49.603169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.603107 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podStartSLOduration=1.603091826 podStartE2EDuration="1.603091826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:49.600300222 +0000 UTC m=+911.067255497" watchObservedRunningTime="2026-11-03 20:18:49.603091826 +0000 UTC m=+911.070047074" Nov 03 20:18:49.692513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.692480 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:49.692823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.692796 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:49.693521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.693492 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:18:49.833236 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:49.833142 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:50.437449 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:50.437412 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:51.096600 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.096561 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr"] Nov 03 20:18:51.102403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.102375 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.129728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.129698 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr"] Nov 03 20:18:51.132744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.132709 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"openshift-service-ca.crt\"" Nov 03 20:18:51.132744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.132730 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"kube-root-ca.crt\"" Nov 03 20:18:51.132926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.132863 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"distinct-w5-fd01ad-kserve-self-signed-certs\"" Nov 03 20:18:51.132926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.132912 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"default-dockercfg-fr8vq\"" Nov 03 20:18:51.192114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.190505 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:51.278404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.278366 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.278404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.278413 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz87g\" (UniqueName: \"kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.278661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.278630 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.278743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.278658 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.278743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.278676 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.278862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.278816 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.340108 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.340053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.381749 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.381798 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz87g\" (UniqueName: \"kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.381985 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.382013 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.382036 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.382108 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.382555 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.383551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.383412 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.384142 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.383924 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.409123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.395935 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.420474 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.420409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.420644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.420519 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz87g\" (UniqueName: \"kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g\") pod \"distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.438151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.436387 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:51.763611 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:51.763570 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr"] Nov 03 20:18:51.765944 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:51.765907 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334cc1ee_0aa4_451f_b1e8_0be38daaff49.slice/crio-61f88290407eeae277f5c43e8bde1018f81ed1f94d6e1893a425af75dfd998bb WatchSource:0}: Error finding container 61f88290407eeae277f5c43e8bde1018f81ed1f94d6e1893a425af75dfd998bb: Status 404 returned error can't find the container with id 61f88290407eeae277f5c43e8bde1018f81ed1f94d6e1893a425af75dfd998bb Nov 03 20:18:52.586712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:52.586667 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" event={"ID":"334cc1ee-0aa4-451f-b1e8-0be38daaff49","Type":"ContainerStarted","Data":"b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238"} Nov 03 20:18:52.587099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:52.586729 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" event={"ID":"334cc1ee-0aa4-451f-b1e8-0be38daaff49","Type":"ContainerStarted","Data":"61f88290407eeae277f5c43e8bde1018f81ed1f94d6e1893a425af75dfd998bb"} Nov 03 20:18:52.628842 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:52.628787 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podStartSLOduration=1.6287704010000001 podStartE2EDuration="1.628770401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:52.626318334 +0000 UTC m=+914.093273589" watchObservedRunningTime="2026-11-03 20:18:52.628770401 +0000 UTC m=+914.095725656" Nov 03 20:18:53.020238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:53.020152 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:53.437377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:53.437337 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:18:53.631655 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:53.631622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:54.596489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:54.596457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:55.830009 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.829968 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn"] Nov 03 20:18:55.835932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.835901 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856487 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856551 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856601 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856629 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856730 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856765 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856802 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gm2q\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856915 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.862859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.856960 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.864883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.864855 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn"] Nov 03 20:18:55.961097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.960999 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961184 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961248 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961303 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961336 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961399 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961424 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961456 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gm2q\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:55.961765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.961590 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.000103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.985688 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.000103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:55.987458 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.000668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.000601 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.004394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.004332 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.013438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.013402 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.013905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.013857 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.015826 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.015776 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.047279 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.047204 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gm2q\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.047940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.047878 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.157105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.156968 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:56.561487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.561363 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:56.567838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.567795 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn"] Nov 03 20:18:56.568063 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:56.567841 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ffd880_d006_42cf_b4e5_93884993ca51.slice/crio-30ed626f942604e10ac4be4453d6fb715fb45fd1073a201edd028a93d269ea9d WatchSource:0}: Error finding container 30ed626f942604e10ac4be4453d6fb715fb45fd1073a201edd028a93d269ea9d: Status 404 returned error can't find the container with id 30ed626f942604e10ac4be4453d6fb715fb45fd1073a201edd028a93d269ea9d Nov 03 20:18:56.571775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.571687 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:56.571877 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.571816 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:56.571877 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.571866 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:56.608401 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.607021 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn"] Nov 03 20:18:56.698494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.698464 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.824107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.824051 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.824425 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.824401 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.824580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.824563 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.824739 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.824718 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.824868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.824852 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.825004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.824988 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2qj\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.825267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.825246 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.825406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.825388 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.832257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.832234 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:56.927519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.924009 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" event={"ID":"a3ffd880-d006-42cf-b4e5-93884993ca51","Type":"ContainerStarted","Data":"30ed626f942604e10ac4be4453d6fb715fb45fd1073a201edd028a93d269ea9d"} Nov 03 20:18:56.927519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:56.924060 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn"] Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.001800 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.001859 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.001963 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.001993 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.002053 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2qj\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.002243 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.002273 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.002324 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.002382 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.003122 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.003923 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.004202 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.004986 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.005432 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.012111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.007291 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.026101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.019476 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.030123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.029898 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.055787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.055521 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2qj\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-t2zhn\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.199173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.199050 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:57.541823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.541790 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn"] Nov 03 20:18:57.544738 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:18:57.544694 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff33112_7b0e_445f_a7c2_d8766dfa2d49.slice/crio-8334451728eb039fb427894e65d6dd710d538ad5e1463b279daf5287e7035ac6 WatchSource:0}: Error finding container 8334451728eb039fb427894e65d6dd710d538ad5e1463b279daf5287e7035ac6: Status 404 returned error can't find the container with id 8334451728eb039fb427894e65d6dd710d538ad5e1463b279daf5287e7035ac6 Nov 03 20:18:57.547678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.547646 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:57.547766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.547720 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:57.547766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.547752 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:57.838882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.838846 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" event={"ID":"cff33112-7b0e-445f-a7c2-d8766dfa2d49","Type":"ContainerStarted","Data":"168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe"} Nov 03 20:18:57.839352 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.838902 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" event={"ID":"cff33112-7b0e-445f-a7c2-d8766dfa2d49","Type":"ContainerStarted","Data":"8334451728eb039fb427894e65d6dd710d538ad5e1463b279daf5287e7035ac6"} Nov 03 20:18:57.841262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.841238 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" event={"ID":"a3ffd880-d006-42cf-b4e5-93884993ca51","Type":"ContainerStarted","Data":"5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb"} Nov 03 20:18:57.901096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.900592 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podStartSLOduration=1.900571651 podStartE2EDuration="1.900571651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:57.882548343 +0000 UTC m=+919.349503630" watchObservedRunningTime="2026-11-03 20:18:57.900571651 +0000 UTC m=+919.367526906" Nov 03 20:18:57.926263 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:57.926207 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podStartSLOduration=2.926192818 podStartE2EDuration="2.926192818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:57.921350829 +0000 UTC m=+919.388306084" watchObservedRunningTime="2026-11-03 20:18:57.926192818 +0000 UTC m=+919.393148071" Nov 03 20:18:58.157662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.157572 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:18:58.159256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.159230 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:58.159344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.159293 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:18:58.202688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.202288 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:18:58.203184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.203157 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:18:58.203369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.203347 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:18:58.265040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.265009 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:58.438529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.438431 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:58.465818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.465790 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:58.862885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:58.862846 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:18:59.157938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:59.157859 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:59.157938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:59.157920 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:18:59.205095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:59.202172 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:18:59.205095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:18:59.202234 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:19:00.132095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:00.131646 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:00.158392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:00.158350 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:19:00.158608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:00.158422 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:19:00.204512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:00.204472 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:19:00.204796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:00.204766 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:19:00.305097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:00.303760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:01.158408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.158367 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:19:01.158791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.158436 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:19:01.206242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.206207 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:19:01.206553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.206503 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:19:01.446107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.445968 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:19:01.458082 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.458045 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:19:01.950174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.950149 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:19:01.972473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:01.972447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:02.159028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:02.158990 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:19:02.159537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:02.159058 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:19:02.164202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:02.164162 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:02.221152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:02.201415 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:19:02.221152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:02.201479 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:19:03.158122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:03.158059 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:19:03.158318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:03.158146 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:19:03.201822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:03.201390 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:19:03.201822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:03.201452 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:19:03.860497 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:03.860404 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:04.161996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:04.161921 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:19:04.162436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:04.162414 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:19:04.163006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:04.162986 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:19:04.200132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:04.200102 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 20:19:04.200259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:04.200161 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 20:19:05.206099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:05.206054 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:19:05.206673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:05.206363 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:19:05.207172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:05.207153 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:19:05.599113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:05.599062 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:05.631115 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:19:05.631063 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:19:05.825920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:05.825874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.318049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.318002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.542060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.542024 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj"] Nov 03 20:19:07.547978 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.547960 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.574293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.574270 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:19:07.574467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.574439 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gwj92\"" Nov 03 20:19:07.574626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.574296 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:19:07.574626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.574279 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:19:07.578861 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.578834 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj"] Nov 03 20:19:07.640459 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.640423 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.640628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.640478 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.640628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.640595 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.640753 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.640716 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk76p\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.640798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.640758 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.640893 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.640873 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.641049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.641033 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.741890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.741858 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.741890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.741895 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.742207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.741913 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.742207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.741949 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.742207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.742049 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk76p\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.742207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.742117 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.742207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.742197 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.742883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.742853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.744607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.744576 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.744706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.744689 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.744812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.744755 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.744986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.744969 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.761771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.761750 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk76p\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.761868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.761812 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9fbxj\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:07.862408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:07.862321 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:08.194192 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:08.194158 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj"] Nov 03 20:19:08.197806 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:19:08.197780 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b2a85c_e89f_4a83_a46e_48895b9ac3dc.slice/crio-964775b3a2bd2c672c6427337e4da2f48ca12f413c9921f86f4f46488cb9a360 WatchSource:0}: Error finding container 964775b3a2bd2c672c6427337e4da2f48ca12f413c9921f86f4f46488cb9a360: Status 404 returned error can't find the container with id 964775b3a2bd2c672c6427337e4da2f48ca12f413c9921f86f4f46488cb9a360 Nov 03 20:19:09.045139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:09.035388 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" event={"ID":"32b2a85c-e89f-4a83-a46e-48895b9ac3dc","Type":"ContainerStarted","Data":"964775b3a2bd2c672c6427337e4da2f48ca12f413c9921f86f4f46488cb9a360"} Nov 03 20:19:09.220096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:09.219276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:11.089695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.089662 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:11.658398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.658356 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp"] Nov 03 20:19:11.667202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.667172 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682390 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682351 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682409 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682428 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682495 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682655 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682542 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682655 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682574 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682655 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682629 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682655 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.682758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.682720 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.706433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.706392 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp"] Nov 03 20:19:11.784232 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784190 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784255 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784278 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784380 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784434 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784471 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784534 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784559 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784632 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.784836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.784716 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.785786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.785519 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.785786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.785736 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.785786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.785749 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.786120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.785251 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.787263 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.787238 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.787660 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.787638 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.804647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.804619 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.805004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.804980 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:11.994304 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:11.994214 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:12.299534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.299495 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:12.299986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.299579 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:12.335862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.335835 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp"] Nov 03 20:19:12.343506 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:19:12.343459 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode535b50e_9c2f_470d_b676_5c9983c4ea65.slice/crio-1d808b3c55f8c35c6c0ae700252718b26ca98722bfc66583d6ec80937ba262d1 WatchSource:0}: Error finding container 1d808b3c55f8c35c6c0ae700252718b26ca98722bfc66583d6ec80937ba262d1: Status 404 returned error can't find the container with id 1d808b3c55f8c35c6c0ae700252718b26ca98722bfc66583d6ec80937ba262d1 Nov 03 20:19:12.346200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.346171 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:12.346274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.346234 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:12.346274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.346263 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:12.719151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:12.719117 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:13.118084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.118020 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" event={"ID":"32b2a85c-e89f-4a83-a46e-48895b9ac3dc","Type":"ContainerStarted","Data":"cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec"} Nov 03 20:19:13.122720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.122692 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:13.125829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.125801 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" event={"ID":"e535b50e-9c2f-470d-b676-5c9983c4ea65","Type":"ContainerStarted","Data":"efda12ec093d2b353b2b0099daa453495592a8a4f905662113b655646b51929a"} Nov 03 20:19:13.126536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.126511 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" event={"ID":"e535b50e-9c2f-470d-b676-5c9983c4ea65","Type":"ContainerStarted","Data":"1d808b3c55f8c35c6c0ae700252718b26ca98722bfc66583d6ec80937ba262d1"} Nov 03 20:19:13.126930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.125592 5043 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9fbxj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:19:13.127800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.127780 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" podUID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:19:13.230093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.226434 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" podStartSLOduration=2.128740727 podStartE2EDuration="6.226411652s" totalImagesPullingTime="4.097670925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:13.222586955 +0000 UTC m=+934.689542234" watchObservedRunningTime="2026-11-03 20:19:13.226411652 +0000 UTC m=+934.693366906" Nov 03 20:19:13.268020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:13.267887 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podStartSLOduration=2.267865493 podStartE2EDuration="2.267865493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:13.266276662 +0000 UTC m=+934.733231919" watchObservedRunningTime="2026-11-03 20:19:13.267865493 +0000 UTC m=+934.734820752" Nov 03 20:19:14.001122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.000885 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:14.001122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.001019 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:19:14.001122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.001064 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:19:14.133890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.133857 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:19:14.339009 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.338973 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:14.998096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.997446 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:19:14.998096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:14.997507 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:19:15.934890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:15.934849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:16.004094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:16.003459 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:19:16.004094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:16.003525 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:19:16.997102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:16.996463 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:19:16.997102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:16.996527 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:19:17.633350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:17.633311 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:17.999494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:17.998408 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:19:17.999494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:17.998473 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:19:18.995381 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:18.995335 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:19:18.995558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:18.995405 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:19:19.220059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:19.220020 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:19.999211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:19.999180 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:19.999695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:19.999672 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:20.000401 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:20.000378 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:19:20.702353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:20.702323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:22.401467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:22.401429 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:23.922767 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:23.922729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:25.411778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:25.411737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:26.702646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.702610 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq"] Nov 03 20:19:26.707595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.707575 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.753234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.753199 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq"] Nov 03 20:19:26.873120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.873056 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.873332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.873180 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.873332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.873224 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.873332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.873275 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.873493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.873387 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.873493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.873453 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszs9\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.876306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.876056 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.876306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.876119 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.876306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.876153 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977164 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977245 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszs9\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977326 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977361 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977391 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977470 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977540 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.977575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977573 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.978028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.977622 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.978110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.978088 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.978349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.978320 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.979497 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.979466 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:26.988894 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.988862 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.003169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:26.991627 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.016145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.005604 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.016145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.006245 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszs9\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.016145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.010861 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.038162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.030081 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-42lfq\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.047486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.047442 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:27.642934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.642822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:27.687222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.663537 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq"] Nov 03 20:19:27.754473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.753519 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:27.754473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.753616 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:27.754473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:27.753663 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:28.186099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:28.174932 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:28.566154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:28.562813 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" event={"ID":"d835a0bf-ebcc-4919-9914-747f20a52fce","Type":"ContainerStarted","Data":"d46b7e06180f22b8afdec10625f931abdc655fc1ccd616be1a781bccfd14c33f"} Nov 03 20:19:29.573182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:29.573149 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" event={"ID":"d835a0bf-ebcc-4919-9914-747f20a52fce","Type":"ContainerStarted","Data":"99cc26dd41e9ef63239980fb4360f12769d50a058409dcced5c750580f7e676f"} Nov 03 20:19:29.627443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:29.627267 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podStartSLOduration=3.627248704 podStartE2EDuration="3.627248704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:29.626331966 +0000 UTC m=+951.093287221" watchObservedRunningTime="2026-11-03 20:19:29.627248704 +0000 UTC m=+951.094203959" Nov 03 20:19:29.929457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:29.929399 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.029627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:30.029590 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.048927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:30.048891 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:30.050548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:30.050515 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:19:30.050697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:30.050575 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 20:19:31.049441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:31.049405 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:19:31.049815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:31.049473 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 20:19:31.723608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:31.723568 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:32.049521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:32.049436 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:19:32.049521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:32.049500 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 20:19:33.049137 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:33.049100 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:19:33.049368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:33.049164 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 20:19:33.404697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:33.404658 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:34.048650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:34.048610 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:19:34.048862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:34.048678 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 20:19:35.006455 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:35.006351 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:35.049183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:35.049146 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:19:35.049359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:35.049208 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 20:19:35.631218 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:19:35.631179 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:19:36.313741 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:36.313708 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:36.315939 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:36.315914 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:36.317501 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:36.317481 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:19:36.619735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:36.619699 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:37.250056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:37.250019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:38.109901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:38.109841 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:39.686993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:39.686953 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:41.331273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:41.331229 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:41.431177 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:41.431138 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:43.085567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:43.085522 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:44.796101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:44.788862 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:45.422015 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:45.421977 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:47.124419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:47.124378 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:48.799379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:48.799301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:50.421135 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:50.421082 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:50.522282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:50.522241 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.135225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:52.135167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:53.630170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:53.630126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:55.004596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:55.004556 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:55.206941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:55.206873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.830296 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:56.830256 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:58.607993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:19:58.607957 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:00.106425 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:00.106374 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:01.740523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:01.740484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:03.330169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:03.330132 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:05.020731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:05.020689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:05.766358 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:20:05.766314 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:20:07.370220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:07.370180 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:20:07.371050 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:07.370763 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:09.008300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:08.990431 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:09.034098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:09.009427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.034098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:09.010292 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 20:20:09.663248 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:09.663204 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:11.125379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:11.125341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.033923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.033894 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9"] Nov 03 20:20:12.037737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.037710 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.073029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.072987 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9"] Nov 03 20:20:12.222013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.221968 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222130 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222171 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222369 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222486 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222569 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222660 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222727 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.222871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.222794 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79x85\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.323916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.323887 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.323945 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324097 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324193 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79x85\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324293 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324396 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324382 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324420 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324530 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324607 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324670 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.324960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.324935 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.325243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.325216 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.325336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.325246 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.325433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.325409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.326807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.326786 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.327220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.327198 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.349057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.349025 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.351243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.351214 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79x85\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.353002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.352979 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:12.686926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.686898 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9"] Nov 03 20:20:12.692604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.692564 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:12.692723 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.692652 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:12.692723 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.692696 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:12.882094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:12.881895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:13.335260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:13.335216 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" event={"ID":"4e635786-b323-443a-a72e-4006154bf7a8","Type":"ContainerStarted","Data":"b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038"} Nov 03 20:20:13.335701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:13.335316 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" event={"ID":"4e635786-b323-443a-a72e-4006154bf7a8","Type":"ContainerStarted","Data":"2f0e6a1cea644c3c8772d519b1fcb70e1fb7daf174a09e527e1df69a51f88a4c"} Nov 03 20:20:13.353994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:13.353963 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:13.355465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:13.355437 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:13.355598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:13.355486 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:13.403624 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:13.403562 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podStartSLOduration=1.403545282 podStartE2EDuration="1.403545282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:13.392628592 +0000 UTC m=+994.859583872" watchObservedRunningTime="2026-11-03 20:20:13.403545282 +0000 UTC m=+994.870500535" Nov 03 20:20:14.353437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:14.353401 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:14.353915 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:14.353457 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:14.744151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:14.744019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:15.353859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:15.353819 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:15.354328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:15.353877 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:16.353424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:16.353387 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:16.353679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:16.353442 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:16.827564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:16.827526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:17.354338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:17.354297 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:17.354561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:17.354375 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:18.353631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:18.353598 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:18.354034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:18.353650 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:18.456198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:18.456164 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:19.354437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:19.354398 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:20:19.354860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:19.354468 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:20:20.199996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:20.199939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:20.529323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:20.529232 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:20.530122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:20.530041 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:20.531210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:20.531181 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:20:21.929846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:21.929799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:23.659089 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:23.659035 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:25.295753 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:25.295715 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.103186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.103145 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx"] Nov 03 20:20:26.108235 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.108196 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.148751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.148713 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx"] Nov 03 20:20:26.266237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266195 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266242 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266280 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnpz\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266398 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266493 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266543 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266671 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266804 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.266921 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.266909 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369247 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369158 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369247 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369274 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369318 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnpz\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369458 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369514 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369560 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369574 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369618 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.369719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369725 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.370308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369849 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.370308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.369928 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.370419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.370331 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.371749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.371724 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.371948 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.371932 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.387354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.387322 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.387480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.387426 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnpz\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.427474 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.427429 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:26.536830 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.536782 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp"] Nov 03 20:20:26.545023 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.545002 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.564453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.564421 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp"] Nov 03 20:20:26.678127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678024 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678305 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678173 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678305 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678264 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678367 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678423 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678492 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678477 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678587 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678556 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppvlj\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678638 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678616 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.678693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.678681 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.757406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.757348 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx"] Nov 03 20:20:26.760477 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:20:26.760440 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4514e4_7a25_4480_8d66_b704218b6098.slice/crio-75300191aa10457e3dfd63a7aa22b26007e196142028fe64cc8c40401575ca0c WatchSource:0}: Error finding container 75300191aa10457e3dfd63a7aa22b26007e196142028fe64cc8c40401575ca0c: Status 404 returned error can't find the container with id 75300191aa10457e3dfd63a7aa22b26007e196142028fe64cc8c40401575ca0c Nov 03 20:20:26.763815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.763782 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:26.763929 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.763862 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:26.763993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.763926 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:26.780046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.780019 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.780340 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.780312 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.780500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.780483 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.780864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.780840 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.780925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.780910 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.780977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.780964 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppvlj\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.781047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.781032 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.781193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.781174 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.781276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.781262 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.784431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.784406 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.785351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.785260 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.785351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.785312 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.785730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.785682 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.785910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.785888 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.787574 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.787550 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.788584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.788492 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.799495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.799465 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.799908 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.799863 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppvlj\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.866670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.866641 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:26.970092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.966988 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd"] Nov 03 20:20:26.977104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:26.976965 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.003973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.003554 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd"] Nov 03 20:20:27.048436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.041849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:27.115182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.112057 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.116311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.116265 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.119493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.116511 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.120326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.120290 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.126691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.124935 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.126691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.125037 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.126691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.125110 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.126691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.125147 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.126691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.125181 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpv6\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.205059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.205008 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp"] Nov 03 20:20:27.208666 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:20:27.208623 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1e8fba_a9bc_4e1d_abd9_99110e67a883.slice/crio-62de5b80743a9c7218419c01636e69f7abf65562d110c6a67b743122b66d8914 WatchSource:0}: Error finding container 62de5b80743a9c7218419c01636e69f7abf65562d110c6a67b743122b66d8914: Status 404 returned error can't find the container with id 62de5b80743a9c7218419c01636e69f7abf65562d110c6a67b743122b66d8914 Nov 03 20:20:27.211814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.211776 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:27.211927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.211850 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:27.211927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.211884 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:27.227473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227429 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227512 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227548 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227581 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227606 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpv6\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227674 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227777 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227826 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227863 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.227889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.227876 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.228231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.228128 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.228700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.228674 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.228817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.228702 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.229123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.229092 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.230931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.230906 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.231189 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.231164 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.248194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.248165 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpv6\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.249141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.249058 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.340474 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.340438 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:27.488373 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.475815 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" event={"ID":"0e1e8fba-a9bc-4e1d-abd9-99110e67a883","Type":"ContainerStarted","Data":"b286e151880344ecebc8883a46232608fe9321abf2f188275531cae89e7fe78c"} Nov 03 20:20:27.488373 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.475886 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" event={"ID":"0e1e8fba-a9bc-4e1d-abd9-99110e67a883","Type":"ContainerStarted","Data":"62de5b80743a9c7218419c01636e69f7abf65562d110c6a67b743122b66d8914"} Nov 03 20:20:27.533950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.533836 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" event={"ID":"4e4514e4-7a25-4480-8d66-b704218b6098","Type":"ContainerStarted","Data":"b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec"} Nov 03 20:20:27.533950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.533901 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" event={"ID":"4e4514e4-7a25-4480-8d66-b704218b6098","Type":"ContainerStarted","Data":"75300191aa10457e3dfd63a7aa22b26007e196142028fe64cc8c40401575ca0c"} Nov 03 20:20:27.558715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.558357 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podStartSLOduration=1.55833981 podStartE2EDuration="1.55833981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:27.556032109 +0000 UTC m=+1009.022987394" watchObservedRunningTime="2026-11-03 20:20:27.55833981 +0000 UTC m=+1009.025295063" Nov 03 20:20:27.616096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.615464 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podStartSLOduration=1.615445287 podStartE2EDuration="1.615445287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:27.614736082 +0000 UTC m=+1009.081691354" watchObservedRunningTime="2026-11-03 20:20:27.615445287 +0000 UTC m=+1009.082400542" Nov 03 20:20:27.677340 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.677310 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd"] Nov 03 20:20:27.731484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.720140 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:27.731484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.720284 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:27.731484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.720365 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:27.867235 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:27.867205 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:28.431942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.431509 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:28.431942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.431644 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:28.431942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.431686 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:28.676418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.674177 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" event={"ID":"a18950f9-3b3c-4610-8f82-67f1eff53bb3","Type":"ContainerStarted","Data":"6d86f3ed698d9910fb82d07f99de8f844637f80a7dace04767d2d10cb9e584da"} Nov 03 20:20:28.676418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.674318 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" event={"ID":"a18950f9-3b3c-4610-8f82-67f1eff53bb3","Type":"ContainerStarted","Data":"8d67c3334dcd231a1043bde8ecdb883864b7469cc65cef47bd521491b0091471"} Nov 03 20:20:28.905981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.855875 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podStartSLOduration=2.855851405 podStartE2EDuration="2.855851405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.758669009 +0000 UTC m=+1010.225624264" watchObservedRunningTime="2026-11-03 20:20:28.855851405 +0000 UTC m=+1010.322806661" Nov 03 20:20:28.965116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.956502 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:28.965116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:28.956560 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:29.068110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.067107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:29.357151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.347154 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:29.357151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.347340 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:29.357151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.347388 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:29.433095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.432587 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:29.433095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.432666 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:29.877097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.869297 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:29.877097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:29.869360 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:30.296163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:30.295457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:30.357686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:30.357652 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:30.357968 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:30.357940 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:30.431532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:30.431492 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:30.431807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:30.431778 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:31.005156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:30.979302 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:31.031227 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.031175 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:31.375093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.346497 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:31.375093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.346559 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:31.432057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.432019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:31.463094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.451208 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:31.463094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.451274 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:31.976095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.930824 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:31.980093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:31.978151 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:32.493216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:32.489475 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:32.493216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:32.489541 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:32.496629 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:32.496595 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:32.496864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:32.496837 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:32.618436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:32.599429 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:33.288097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.269710 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:33.288097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.269774 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:33.429967 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.429926 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:33.577855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.577805 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:33.653716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.445357 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:33.654156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.654120 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:33.872986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.872880 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:33.873388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:33.873332 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:34.341872 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.341831 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:34.342169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.342139 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:34.430970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.430920 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:20:34.431314 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.431269 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:20:34.725154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.723884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:34.878732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.878638 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:20:34.879055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:34.879003 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:20:35.341620 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:35.341580 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:20:35.342310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:35.342271 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 20:20:35.550617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:35.550582 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:35.551821 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:35.551794 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:35.553496 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:35.553476 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:20:35.675203 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:20:35.675048 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:20:36.133033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.132980 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:36.133829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.133805 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:36.135215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.135182 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:20:36.476197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.476108 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.589139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.588980 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:36.591018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.590993 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:36.591911 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:36.591835 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:20:38.211440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:38.211404 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:40.018593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:40.018542 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:41.727099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.726509 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd"] Nov 03 20:20:41.767098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.766922 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.818099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.812100 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd"] Nov 03 20:20:41.928872 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.928824 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.929192 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.929168 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.929356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.929336 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.929555 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.929534 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.929825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.929709 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9d4d\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.929980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.929960 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.933881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.933853 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.934139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.934119 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:41.935583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:41.935556 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067027 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067192 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067417 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067476 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067536 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067627 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067720 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9d4d\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.067782 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.068026 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.198530 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.256316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.198753 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.319109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.258899 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.457333 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.330895 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.546127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.546026 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.557171 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.549132 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv"] Nov 03 20:20:42.635494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.635453 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.640482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.640450 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.671098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.659844 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.694116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.680838 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9d4d\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:42.714297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.714253 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:42.838097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.834453 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:43.041272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.902195 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv"] Nov 03 20:20:43.041569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.977220 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:20:43.041771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:42.930171 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.042139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.042116 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.042360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.042324 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.042605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.042579 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg426\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.069242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.068263 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.069242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.068339 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.069242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.068465 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.069242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.068547 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.069242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.068575 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.203830 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.203889 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.203950 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.204093 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.204148 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.204243 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg426\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.204390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.204436 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.204531 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.205505 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.205799 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.208153 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.215178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.208436 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.260273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.260233 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.293931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.273846 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.649109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.641485 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.742818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.742763 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:43.882477 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:43.882437 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 20:20:44.118307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:44.118250 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:44.125290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:44.125242 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:20:44.346514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:44.346061 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:44.378432 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:44.348323 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg426\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:44.638695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:44.638372 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:45.118102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:45.113367 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 20:20:45.155096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:45.119606 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:20:45.662592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:45.662305 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:20:45.757973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:45.757924 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd"] Nov 03 20:20:45.775106 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:20:45.768808 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb39e4e7_e05b_4abd_b40f_24a79aeb7696.slice/crio-d3adfe6e1013a0cac5ab98029f32f5037770c41c3babb7352289cd0cd4c2b599 WatchSource:0}: Error finding container d3adfe6e1013a0cac5ab98029f32f5037770c41c3babb7352289cd0cd4c2b599: Status 404 returned error can't find the container with id d3adfe6e1013a0cac5ab98029f32f5037770c41c3babb7352289cd0cd4c2b599 Nov 03 20:20:45.936091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:45.918934 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:20:46.060349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.059917 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:20:46.105141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.102270 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:46.105141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.102374 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:46.105141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.102451 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:46.593007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.592969 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" event={"ID":"eb39e4e7-e05b-4abd-b40f-24a79aeb7696","Type":"ContainerStarted","Data":"d3adfe6e1013a0cac5ab98029f32f5037770c41c3babb7352289cd0cd4c2b599"} Nov 03 20:20:46.834525 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:20:46.834490 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05af44cb_7d4f_4736_a300_62899d4a4fef.slice/crio-9c8b6b7af78aec77c85ef9669497e0551548c5d341d4bbd441d38a11f4d26d02 WatchSource:0}: Error finding container 9c8b6b7af78aec77c85ef9669497e0551548c5d341d4bbd441d38a11f4d26d02: Status 404 returned error can't find the container with id 9c8b6b7af78aec77c85ef9669497e0551548c5d341d4bbd441d38a11f4d26d02 Nov 03 20:20:46.834718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.834563 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv"] Nov 03 20:20:46.838187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.838145 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:46.838286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.838254 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:46.838336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:46.838315 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:47.291964 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.291729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:47.627561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.624140 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" event={"ID":"05af44cb-7d4f-4736-a300-62899d4a4fef","Type":"ContainerStarted","Data":"49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221"} Nov 03 20:20:47.627561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.624212 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" event={"ID":"05af44cb-7d4f-4736-a300-62899d4a4fef","Type":"ContainerStarted","Data":"9c8b6b7af78aec77c85ef9669497e0551548c5d341d4bbd441d38a11f4d26d02"} Nov 03 20:20:47.627561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.626595 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" event={"ID":"eb39e4e7-e05b-4abd-b40f-24a79aeb7696","Type":"ContainerStarted","Data":"5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc"} Nov 03 20:20:47.641868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.641823 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:47.643936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.643908 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:47.644089 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.643966 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:47.720095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.719891 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podStartSLOduration=5.71987014 podStartE2EDuration="5.71987014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:47.674336779 +0000 UTC m=+1029.141292053" watchObservedRunningTime="2026-11-03 20:20:47.71987014 +0000 UTC m=+1029.186825395" Nov 03 20:20:47.741408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.739896 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podStartSLOduration=6.73987448 podStartE2EDuration="6.73987448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:47.718050916 +0000 UTC m=+1029.185006175" watchObservedRunningTime="2026-11-03 20:20:47.73987448 +0000 UTC m=+1029.206829734" Nov 03 20:20:47.903431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.892372 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:47.903431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.893704 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:47.903431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:47.893758 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:48.656542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:48.655791 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:48.656542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:48.655852 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:48.900097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:48.898571 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:48.900097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:48.898632 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:49.242163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:49.242122 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:49.870096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:49.868480 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:49.870096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:49.868562 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:49.899146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:49.888891 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:49.899146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:49.888951 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:50.643567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:50.643526 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:50.643975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:50.643828 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:50.918189 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:50.917889 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:50.918189 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:50.917950 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:51.231672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:51.231554 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.645452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:51.645417 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:51.645698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:51.645675 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:51.889216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:51.884973 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:51.889216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:51.885032 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:52.643542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:52.643499 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:52.644156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:52.644121 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:52.953630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:52.953516 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:52.953951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:52.953920 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:53.198565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:53.198416 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:53.641538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:53.641490 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:20:53.641729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:53.641576 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 20:20:53.883184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:53.883145 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:20:53.884317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:53.884267 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:20:54.828814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:54.828772 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:54.829270 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:54.829238 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:54.830432 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:54.830372 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:20:54.886456 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:54.886405 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:54.886909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:54.886754 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:54.887605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:54.887585 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:20:55.017126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:55.015862 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.721727 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:56.721383 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:58.302569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:58.302533 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:59.915827 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:20:59.915793 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:01.504858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:01.504817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:03.206147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:03.206086 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:04.862839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:04.862798 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:05.631557 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:21:05.631524 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:21:06.427557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:06.427444 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:08.048215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:08.048170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:09.595868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:09.595828 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:11.221782 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:11.221743 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:16.414097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:16.344951 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:16.435128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:16.429424 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:17.442170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:17.433611 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:17.444565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:17.433678 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 20:21:18.455580 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:21:18.449785 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.756s" Nov 03 20:21:18.486291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:18.486259 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:21:18.486563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:18.486547 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:21:18.501097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:18.452718 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:18.523148 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:18.452784 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:21:19.227626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:19.227588 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.493151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:19.492197 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:19.508213 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:19.508145 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:19.509626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:19.509578 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:19.549657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:19.549407 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:21:19.549915 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:19.549891 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:21:21.008957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:21.008918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:22.664816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:22.664773 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:22.767540 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:22.767498 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:24.410190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:24.410158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.912280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:25.912231 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:27.524570 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:27.524532 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:29.206166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:29.206127 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:30.365650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:30.365611 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:30.847460 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:30.847427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:32.624448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:32.624411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.915601 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:33.915556 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.022598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:34.022562 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.181000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:34.180913 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.294646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:34.293050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.550820 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:34.550723 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.650592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:34.650549 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:35.128720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:35.128681 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:35.632264 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:21:35.632226 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:21:36.097315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:36.097278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:37.845457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:37.845423 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:37.960970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:37.960931 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:38.062007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:38.061968 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.757347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:39.757306 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.838215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:40.838174 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.943484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:40.943447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.114577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:41.114459 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.377175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:41.377050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.566124 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:41.566093 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.987963 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:41.987804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:42.858639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:42.858602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:42.974688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:42.974651 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:43.141973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:43.141881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:43.276981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:43.276874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:44.606469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:44.606423 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:44.725518 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:44.701538 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 20:21:45.734554 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:45.734468 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:45.737199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:45.735951 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:21:45.761344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:45.756119 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:21:45.761344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:45.760447 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:21:46.581228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:46.581190 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:21:46.875325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:46.875221 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:21:47.601952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:47.601903 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:47.917255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:47.917145 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:48.598878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:48.598836 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:48.628232 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:48.628185 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:48.653238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:48.653197 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:21:48.917169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:48.911465 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:48.917169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:48.911527 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:48.977266 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:48.953144 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:21:49.144324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.040320 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.195124 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.195010 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:49.287770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.287727 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:49.311657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.291758 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:49.311931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.311895 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:49.312058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.246188 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:49.312203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.312182 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:49.312303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.286931 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:49.312428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.312409 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:49.467801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.467700 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded" Nov 03 20:21:49.495636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.495598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.640849 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.640803 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:49.795553 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:21:49.741796 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.065s" Nov 03 20:21:49.832203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.818473 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:49.836287 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:49.836222 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:50.171673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:50.171620 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:50.848096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:50.765084 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:50.873878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:50.873831 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:50.882106 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:50.878701 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 20:21:51.052693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:50.783239 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:51.177379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.177271 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:51.251342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.064338 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:51.251342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.225846 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:51.251342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.138337 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:51.251342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.225937 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:51.529431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.529282 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:21:51.615396 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.615350 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:51.615717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.615681 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:51.940608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:51.940561 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:52.131104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.130125 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:52.131104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.130189 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:52.342269 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.342219 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:52.342775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.342298 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:52.369109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.369049 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-42lfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:52.409895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.409832 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:52.530220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.530181 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:52.576348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.576301 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:21:52.793686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:52.774032 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:21:53.101546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:53.101494 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:53.816517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:53.816469 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:54.439643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:54.439604 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:54.548853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:54.548817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:55.419280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:55.419238 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:55.520507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:55.520475 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.914141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:56.914043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:58.574903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:58.574861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:58.675773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:21:58.675738 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:00.043057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.043012 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7"] Nov 03 20:22:00.048064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.048043 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.074687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.074657 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9798ad8e-serving-cert\"" Nov 03 20:22:00.083449 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.083417 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7"] Nov 03 20:22:00.129816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.129766 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:00.208519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.208486 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.208700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.208665 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.208758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.208741 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.208930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.208910 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n568p\" (UniqueName: \"kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.226623 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.226588 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:00.311851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.310119 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n568p\" (UniqueName: \"kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.311851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.310266 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.311851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.310429 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.311851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.310531 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.311851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.311444 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.316462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.316430 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.336260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.329597 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.338010 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.333682 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n568p\" (UniqueName: \"kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p\") pod \"maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.362730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.362692 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:00.690229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.690098 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7"] Nov 03 20:22:00.693921 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:22:00.693884 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bf7928_aa74_49da_8dba_5f2c154a2483.slice/crio-4ae28bd8fdbfa0955c58674ec31273ecba143be2cb461e7fa9546708380ebb67 WatchSource:0}: Error finding container 4ae28bd8fdbfa0955c58674ec31273ecba143be2cb461e7fa9546708380ebb67: Status 404 returned error can't find the container with id 4ae28bd8fdbfa0955c58674ec31273ecba143be2cb461e7fa9546708380ebb67 Nov 03 20:22:00.695836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:00.695813 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:22:01.229290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:01.229246 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" event={"ID":"a8bf7928-aa74-49da-8dba-5f2c154a2483","Type":"ContainerStarted","Data":"4ae28bd8fdbfa0955c58674ec31273ecba143be2cb461e7fa9546708380ebb67"} Nov 03 20:22:01.701979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:01.701943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:02.239939 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:02.239900 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" event={"ID":"a8bf7928-aa74-49da-8dba-5f2c154a2483","Type":"ContainerStarted","Data":"303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7"} Nov 03 20:22:02.240471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:02.240023 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:02.282694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:02.282643 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" podStartSLOduration=1.953284048 podStartE2EDuration="2.282628841s" totalImagesPullingTime="329.344793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:02.279300808 +0000 UTC m=+1103.746256061" watchObservedRunningTime="2026-11-03 20:22:02.282628841 +0000 UTC m=+1103.749584094" Nov 03 20:22:03.313421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:03.313382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.286645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:04.286596 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.388567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:04.388529 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.504998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:04.504956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.771035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:04.770996 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.921247 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:04.921158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.211363 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:05.211327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.461323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:05.461287 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.560281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:05.560195 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.652264 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:22:05.652226 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:22:05.671601 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:05.670902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:06.963958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:06.963923 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7"] Nov 03 20:22:06.969749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:06.969728 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:06.995581 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:06.995550 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7"] Nov 03 20:22:06.995755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:06.995739 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-11a7d765-serving-cert\"" Nov 03 20:22:07.046197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.046155 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.046379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.046210 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.046379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.046335 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.046517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.046387 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfpq\" (UniqueName: \"kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.047435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.047393 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.132730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.132677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.148088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.148033 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.148255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.148098 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.148255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.148222 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.148382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.148277 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfpq\" (UniqueName: \"kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.149013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.148989 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.150685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.150661 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.150805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.150767 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.168744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.168719 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfpq\" (UniqueName: \"kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq\") pod \"maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.233264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.233185 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.295108 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.294842 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:07.479912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.479872 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.589177 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.589139 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.640313 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:07.640288 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7"] Nov 03 20:22:07.645469 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:22:07.645440 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988642c0_6be0_4238_9e5d_9181b56930ba.slice/crio-60468f693b91482410eb12404b3904efc46cbf4a8da324574fa14a38518f861f WatchSource:0}: Error finding container 60468f693b91482410eb12404b3904efc46cbf4a8da324574fa14a38518f861f: Status 404 returned error can't find the container with id 60468f693b91482410eb12404b3904efc46cbf4a8da324574fa14a38518f861f Nov 03 20:22:08.270152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:08.265813 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:08.376626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:08.376583 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" event={"ID":"988642c0-6be0-4238-9e5d-9181b56930ba","Type":"ContainerStarted","Data":"4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71"} Nov 03 20:22:08.376812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:08.376652 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" event={"ID":"988642c0-6be0-4238-9e5d-9181b56930ba","Type":"ContainerStarted","Data":"60468f693b91482410eb12404b3904efc46cbf4a8da324574fa14a38518f861f"} Nov 03 20:22:08.378886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:08.377833 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:08.421094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:08.416093 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" podStartSLOduration=2.072054135 podStartE2EDuration="2.416056049s" totalImagesPullingTime="344.001914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:08.414631804 +0000 UTC m=+1109.881587058" watchObservedRunningTime="2026-11-03 20:22:08.416056049 +0000 UTC m=+1109.883011304" Nov 03 20:22:09.000153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:09.000108 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:09.189662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:09.189630 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:09.288534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:09.288430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.609096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:10.604712 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.730095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:10.723830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:12.252616 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:12.252569 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.804028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:13.803992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:15.292358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:15.292317 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:15.400173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:15.400139 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:16.217290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:16.217252 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:16.317728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:16.317688 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:16.532658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:16.532484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:16.911402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:16.911362 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:17.011602 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:17.011569 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:17.885528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:17.885445 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.346398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:19.346355 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.443266 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:19.443212 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:19.805792 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:19.805697 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:19.914227 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:19.914185 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:22:20.064702 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:20.064610 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:20.065043 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:20.065011 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:20.769099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:20.766011 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:20.769099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:20.766096 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:20.808675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:20.808629 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:20.928257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:20.914797 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 03 20:22:21.254292 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:21.213684 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:22:21.302829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:21.302796 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:22:21.810733 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:21.810675 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:22.366094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:22.360110 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:22:22.366396 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:22.366376 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:22:22.396735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:22.394733 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:22.396735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:22.394810 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:22.549099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:22.533000 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Nov 03 20:22:23.092040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.091715 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:23.092040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.091777 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:23.228557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.228519 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:23.400554 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.400459 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:23.400727 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.400548 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:23.822159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.821133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:23.940862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:23.938543 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.034443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:24.034403 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.261970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:24.261934 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.916262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:24.916218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:25.375912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:25.375861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.507818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.507781 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2"] Nov 03 20:22:26.514246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.514216 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.530528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.530506 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1b67df53-serving-cert\"" Nov 03 20:22:26.546317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.546289 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2"] Nov 03 20:22:26.591670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.591637 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.641487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.641445 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrldv\" (UniqueName: \"kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.641679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.641577 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.641679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.641636 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.641820 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.641716 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.692812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.692751 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.743297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.743271 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrldv\" (UniqueName: \"kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.743473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.743331 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.743473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.743370 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.743594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.743474 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.744195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.744162 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.746036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.746015 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.746244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.746221 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.763234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.763184 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrldv\" (UniqueName: \"kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv\") pod \"maas-api-e2e-ait-1b67df53-6867f56759-8x9p2\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:26.804515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.804427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.834393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:26.834348 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:27.198529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:27.198497 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2"] Nov 03 20:22:27.200189 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:22:27.200154 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edd320a_816a_485d_bb86_5faa0b314b2f.slice/crio-a6e37c1a2e5a38e4547e812db60384025a606aa160594e53ed97c6d22538d491 WatchSource:0}: Error finding container a6e37c1a2e5a38e4547e812db60384025a606aa160594e53ed97c6d22538d491: Status 404 returned error can't find the container with id a6e37c1a2e5a38e4547e812db60384025a606aa160594e53ed97c6d22538d491 Nov 03 20:22:27.273806 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:27.273767 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" event={"ID":"4edd320a-816a-485d-bb86-5faa0b314b2f","Type":"ContainerStarted","Data":"a6e37c1a2e5a38e4547e812db60384025a606aa160594e53ed97c6d22538d491"} Nov 03 20:22:28.283243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:28.283197 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" event={"ID":"4edd320a-816a-485d-bb86-5faa0b314b2f","Type":"ContainerStarted","Data":"a5cc561b1af150d99ffdba3d49bb6dab3642545f49a63a70baf1408ae21c7cc0"} Nov 03 20:22:28.283739 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:28.283299 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:28.321523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:28.321465 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" podStartSLOduration=1.919055378 podStartE2EDuration="2.321449076s" totalImagesPullingTime="402.393698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:28.319450197 +0000 UTC m=+1129.786405452" watchObservedRunningTime="2026-11-03 20:22:28.321449076 +0000 UTC m=+1129.788404329" Nov 03 20:22:28.398862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:28.398827 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.254097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:30.242544 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.340559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:30.340525 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.531011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:30.528673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:32.167326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:32.167290 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:33.916088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:33.916045 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.194845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:34.194753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.305797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:34.305053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.315559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:34.315526 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:22:35.602401 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:35.602355 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:35.634365 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:22:35.634325 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:22:37.210149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:37.210112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.460541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:38.460497 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7"] Nov 03 20:22:38.461218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:38.461187 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" podUID="a8bf7928-aa74-49da-8dba-5f2c154a2483" containerName="maas-api" containerID="cri-o://303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7" gracePeriod=30 Nov 03 20:22:38.516640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:38.516598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.941280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:38.941246 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:38.995408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:38.995366 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.032193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.032157 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls\") pod \"a8bf7928-aa74-49da-8dba-5f2c154a2483\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " Nov 03 20:22:39.032393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.032234 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs\") pod \"a8bf7928-aa74-49da-8dba-5f2c154a2483\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " Nov 03 20:22:39.032470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.032409 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p\" (UniqueName: \"kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p\") pod \"a8bf7928-aa74-49da-8dba-5f2c154a2483\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " Nov 03 20:22:39.032622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.032555 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca\") pod \"a8bf7928-aa74-49da-8dba-5f2c154a2483\" (UID: \"a8bf7928-aa74-49da-8dba-5f2c154a2483\") " Nov 03 20:22:39.033352 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.033319 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca" pod "a8bf7928-aa74-49da-8dba-5f2c154a2483" (UID: "a8bf7928-aa74-49da-8dba-5f2c154a2483"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:39.035552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.035505 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs" pod "a8bf7928-aa74-49da-8dba-5f2c154a2483" (UID: "a8bf7928-aa74-49da-8dba-5f2c154a2483"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:39.035825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.035796 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p" pod "a8bf7928-aa74-49da-8dba-5f2c154a2483" (UID: "a8bf7928-aa74-49da-8dba-5f2c154a2483"). InnerVolumeSpecName "kube-api-access-n568p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:39.035917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.035832 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls" pod "a8bf7928-aa74-49da-8dba-5f2c154a2483" (UID: "a8bf7928-aa74-49da-8dba-5f2c154a2483"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:39.123212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.123176 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.135538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.135479 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n568p\" (UniqueName: \"kubernetes.io/projected/a8bf7928-aa74-49da-8dba-5f2c154a2483-kube-api-access-n568p\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:39.135538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.135504 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf7928-aa74-49da-8dba-5f2c154a2483-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:39.135538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.135517 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:39.135538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.135526 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8bf7928-aa74-49da-8dba-5f2c154a2483-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:39.208443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.208391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.415812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.415717 5043 generic.go:356] "Generic (PLEG): container finished" podID="a8bf7928-aa74-49da-8dba-5f2c154a2483" containerID="303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7" exitCode=0 Nov 03 20:22:39.415989 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.415860 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" event={"ID":"a8bf7928-aa74-49da-8dba-5f2c154a2483","Type":"ContainerDied","Data":"303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7"} Nov 03 20:22:39.415989 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.415911 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" event={"ID":"a8bf7928-aa74-49da-8dba-5f2c154a2483","Type":"ContainerDied","Data":"4ae28bd8fdbfa0955c58674ec31273ecba143be2cb461e7fa9546708380ebb67"} Nov 03 20:22:39.415989 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.415938 5043 scope.go:122] "RemoveContainer" containerID="303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7" Nov 03 20:22:39.416201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.416112 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7" Nov 03 20:22:39.441993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.441959 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.443336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.443319 5043 scope.go:122] "RemoveContainer" containerID="303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7" Nov 03 20:22:39.443588 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:22:39.443564 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7\": container with ID starting with 303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7 not found: ID does not exist" containerID="303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7" Nov 03 20:22:39.443681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.443596 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7"} err="failed to get container status \"303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7\": rpc error: code = NotFound desc = could not find container \"303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7\": container with ID starting with 303f26a5556bae6d5bc6fe88251b74dad7b822376e41430f8c8107961bf007f7 not found: ID does not exist" Nov 03 20:22:39.485788 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.477338 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7"] Nov 03 20:22:39.501094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.497124 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9798ad8e-57ccbb94df-2ddm7"] Nov 03 20:22:39.933770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:39.933729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:40.385006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:40.384969 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:40.530100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:40.529417 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:40.683311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:40.683229 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bf7928-aa74-49da-8dba-5f2c154a2483" path="/var/lib/kubelet/pods/a8bf7928-aa74-49da-8dba-5f2c154a2483/volumes" Nov 03 20:22:41.159641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:41.159587 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.881144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:42.881107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:43.552782 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:43.552733 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:43.652021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:43.651978 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:43.751134 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:43.751098 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:44.889127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:44.889040 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.804198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:45.804001 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.901831 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:45.901787 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:46.449602 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:46.449567 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7"] Nov 03 20:22:46.450041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:46.449988 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" podUID="988642c0-6be0-4238-9e5d-9181b56930ba" containerName="maas-api" containerID="cri-o://4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71" gracePeriod=30 Nov 03 20:22:46.509834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:46.509806 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:46.910134 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:46.909041 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:47.020136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.014629 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca\") pod \"988642c0-6be0-4238-9e5d-9181b56930ba\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " Nov 03 20:22:47.020136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.014785 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq\" (UniqueName: \"kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq\") pod \"988642c0-6be0-4238-9e5d-9181b56930ba\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " Nov 03 20:22:47.020136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.014887 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs\") pod \"988642c0-6be0-4238-9e5d-9181b56930ba\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " Nov 03 20:22:47.020136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.015223 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls\") pod \"988642c0-6be0-4238-9e5d-9181b56930ba\" (UID: \"988642c0-6be0-4238-9e5d-9181b56930ba\") " Nov 03 20:22:47.020136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.018151 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca" pod "988642c0-6be0-4238-9e5d-9181b56930ba" (UID: "988642c0-6be0-4238-9e5d-9181b56930ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:47.020672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.020639 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs" pod "988642c0-6be0-4238-9e5d-9181b56930ba" (UID: "988642c0-6be0-4238-9e5d-9181b56930ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:47.026854 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.026799 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls" pod "988642c0-6be0-4238-9e5d-9181b56930ba" (UID: "988642c0-6be0-4238-9e5d-9181b56930ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:47.044009 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.043962 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq" pod "988642c0-6be0-4238-9e5d-9181b56930ba" (UID: "988642c0-6be0-4238-9e5d-9181b56930ba"). InnerVolumeSpecName "kube-api-access-vxfpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:47.117687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.117646 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:47.117687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.117685 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/988642c0-6be0-4238-9e5d-9181b56930ba-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:47.117687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.117695 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfpq\" (UniqueName: \"kubernetes.io/projected/988642c0-6be0-4238-9e5d-9181b56930ba-kube-api-access-vxfpq\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:47.118038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.117705 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/988642c0-6be0-4238-9e5d-9181b56930ba-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:22:47.331896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.331865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.442179 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.442132 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.521565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.520038 5043 generic.go:356] "Generic (PLEG): container finished" podID="988642c0-6be0-4238-9e5d-9181b56930ba" containerID="4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71" exitCode=0 Nov 03 20:22:47.521565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.520305 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" event={"ID":"988642c0-6be0-4238-9e5d-9181b56930ba","Type":"ContainerDied","Data":"4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71"} Nov 03 20:22:47.521565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.520365 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" event={"ID":"988642c0-6be0-4238-9e5d-9181b56930ba","Type":"ContainerDied","Data":"60468f693b91482410eb12404b3904efc46cbf4a8da324574fa14a38518f861f"} Nov 03 20:22:47.521565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.520398 5043 scope.go:122] "RemoveContainer" containerID="4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71" Nov 03 20:22:47.521565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.520585 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7" Nov 03 20:22:47.541589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.541552 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.545039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.545021 5043 scope.go:122] "RemoveContainer" containerID="4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71" Nov 03 20:22:47.545375 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:22:47.545355 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71\": container with ID starting with 4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71 not found: ID does not exist" containerID="4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71" Nov 03 20:22:47.545436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.545388 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71"} err="failed to get container status \"4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71\": rpc error: code = NotFound desc = could not find container \"4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71\": container with ID starting with 4d93ca6fd9a4dc3ef5729b4c0b0b59fb40911942c047770a0a67995ed0ecec71 not found: ID does not exist" Nov 03 20:22:47.574611 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.574583 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7"] Nov 03 20:22:47.588265 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:47.588189 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11a7d765-7cdfdf6775-trqd7"] Nov 03 20:22:48.257490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:48.256208 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:48.685202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:48.685157 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988642c0-6be0-4238-9e5d-9181b56930ba" path="/var/lib/kubelet/pods/988642c0-6be0-4238-9e5d-9181b56930ba/volumes" Nov 03 20:22:48.839777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:48.839746 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:49.251846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:49.251809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:49.369592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:49.369472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:49.572223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:49.559128 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.204741 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:51.204709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:52.522820 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:52.522782 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:52.623229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:52.623194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:53.184518 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:53.184483 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:54.467494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:54.467455 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:54.601831 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:54.601793 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:55.059066 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:55.059027 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:57.049994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:57.049935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.697596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:58.697561 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.438778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:59.438734 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.545949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:22:59.545907 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.414824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.414792 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm"] Nov 03 20:23:00.415863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.415830 5043 cpu_manager.go:414] "containerMap: removing container" podUID="988642c0-6be0-4238-9e5d-9181b56930ba" containerName="maas-api" Nov 03 20:23:00.415990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.415870 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988642c0-6be0-4238-9e5d-9181b56930ba" containerName="maas-api" Nov 03 20:23:00.415990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.415901 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a8bf7928-aa74-49da-8dba-5f2c154a2483" containerName="maas-api" Nov 03 20:23:00.415990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.415912 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bf7928-aa74-49da-8dba-5f2c154a2483" containerName="maas-api" Nov 03 20:23:00.416240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.416222 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="988642c0-6be0-4238-9e5d-9181b56930ba" containerName="maas-api" Nov 03 20:23:00.416297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.416245 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bf7928-aa74-49da-8dba-5f2c154a2483" containerName="maas-api" Nov 03 20:23:00.420839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.420816 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.439943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.439913 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm"] Nov 03 20:23:00.445515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.445490 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-04bf5fc9-serving-cert\"" Nov 03 20:23:00.513036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.504435 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.530643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.530608 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.530850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.530788 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.530913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.530888 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvpj\" (UniqueName: \"kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.531058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.531044 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.599844 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.599808 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.632910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.632877 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvpj\" (UniqueName: \"kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.633101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.633082 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.633212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.633195 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.633315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.633302 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.634033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.634002 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.635543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.635519 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.635646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.635593 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.650811 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.650791 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvpj\" (UniqueName: \"kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj\") pod \"maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:00.711146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.710804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.736928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:00.736899 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:01.083386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:01.083358 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm"] Nov 03 20:23:01.086781 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:01.086745 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1070ca08_7ee4_4883_bb63_c8fd043226ae.slice/crio-57dbd59ac7d291862761af698d99ad9f08e302256759c640998b4eb1714f98a9 WatchSource:0}: Error finding container 57dbd59ac7d291862761af698d99ad9f08e302256759c640998b4eb1714f98a9: Status 404 returned error can't find the container with id 57dbd59ac7d291862761af698d99ad9f08e302256759c640998b4eb1714f98a9 Nov 03 20:23:01.825692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:01.825656 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" event={"ID":"1070ca08-7ee4-4883-bb63-c8fd043226ae","Type":"ContainerStarted","Data":"d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655"} Nov 03 20:23:01.826355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:01.825718 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" event={"ID":"1070ca08-7ee4-4883-bb63-c8fd043226ae","Type":"ContainerStarted","Data":"57dbd59ac7d291862761af698d99ad9f08e302256759c640998b4eb1714f98a9"} Nov 03 20:23:01.826355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:01.826199 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:01.868462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:01.868409 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" podStartSLOduration=1.503330976 podStartE2EDuration="1.86839489s" totalImagesPullingTime="365.063914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:01.864951395 +0000 UTC m=+1163.331906649" watchObservedRunningTime="2026-11-03 20:23:01.86839489 +0000 UTC m=+1163.335350144" Nov 03 20:23:02.296189 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:02.296056 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:03.568366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:03.568325 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2"] Nov 03 20:23:03.568848 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:03.568627 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" podUID="4edd320a-816a-485d-bb86-5faa0b314b2f" containerName="maas-api" containerID="cri-o://a5cc561b1af150d99ffdba3d49bb6dab3642545f49a63a70baf1408ae21c7cc0" gracePeriod=30 Nov 03 20:23:03.641651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:03.641608 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.004147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.004041 5043 generic.go:356] "Generic (PLEG): container finished" podID="4edd320a-816a-485d-bb86-5faa0b314b2f" containerID="a5cc561b1af150d99ffdba3d49bb6dab3642545f49a63a70baf1408ae21c7cc0" exitCode=0 Nov 03 20:23:04.004402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.004336 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" event={"ID":"4edd320a-816a-485d-bb86-5faa0b314b2f","Type":"ContainerDied","Data":"a5cc561b1af150d99ffdba3d49bb6dab3642545f49a63a70baf1408ae21c7cc0"} Nov 03 20:23:04.004402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.004387 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" event={"ID":"4edd320a-816a-485d-bb86-5faa0b314b2f","Type":"ContainerDied","Data":"a6e37c1a2e5a38e4547e812db60384025a606aa160594e53ed97c6d22538d491"} Nov 03 20:23:04.004593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.004410 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e37c1a2e5a38e4547e812db60384025a606aa160594e53ed97c6d22538d491" Nov 03 20:23:04.029044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.029013 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:23:04.064807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.064775 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs\") pod \"4edd320a-816a-485d-bb86-5faa0b314b2f\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " Nov 03 20:23:04.065172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.064828 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca\") pod \"4edd320a-816a-485d-bb86-5faa0b314b2f\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " Nov 03 20:23:04.065172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.064878 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv\" (UniqueName: \"kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv\") pod \"4edd320a-816a-485d-bb86-5faa0b314b2f\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " Nov 03 20:23:04.065172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.064990 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls\") pod \"4edd320a-816a-485d-bb86-5faa0b314b2f\" (UID: \"4edd320a-816a-485d-bb86-5faa0b314b2f\") " Nov 03 20:23:04.065428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.065397 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca" pod "4edd320a-816a-485d-bb86-5faa0b314b2f" (UID: "4edd320a-816a-485d-bb86-5faa0b314b2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:04.065947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.065921 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edd320a-816a-485d-bb86-5faa0b314b2f-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:04.067996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.067964 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls" pod "4edd320a-816a-485d-bb86-5faa0b314b2f" (UID: "4edd320a-816a-485d-bb86-5faa0b314b2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:04.077796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.067829 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs" pod "4edd320a-816a-485d-bb86-5faa0b314b2f" (UID: "4edd320a-816a-485d-bb86-5faa0b314b2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:04.094901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.092279 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv" pod "4edd320a-816a-485d-bb86-5faa0b314b2f" (UID: "4edd320a-816a-485d-bb86-5faa0b314b2f"). InnerVolumeSpecName "kube-api-access-jrldv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:04.167260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.167229 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:04.167260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.167256 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4edd320a-816a-485d-bb86-5faa0b314b2f-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:04.167260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.167265 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrldv\" (UniqueName: \"kubernetes.io/projected/4edd320a-816a-485d-bb86-5faa0b314b2f-kube-api-access-jrldv\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:04.212063 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.211700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.343220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.343180 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.446892 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.445903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.559670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:04.559563 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:05.023277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:05.021769 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2" Nov 03 20:23:05.076903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:05.076869 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2"] Nov 03 20:23:05.090290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:05.090256 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b67df53-6867f56759-8x9p2"] Nov 03 20:23:05.631190 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:23:05.631154 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:23:05.776527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:05.776494 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.249218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:06.249172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.347800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:06.347765 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.448127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:06.443688 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.680648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:06.680606 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4edd320a-816a-485d-bb86-5faa0b314b2f" path="/var/lib/kubelet/pods/4edd320a-816a-485d-bb86-5faa0b314b2f/volumes" Nov 03 20:23:07.905548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:07.905515 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:08.115274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:08.115229 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.774999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:09.774961 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.934881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:09.934767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.054963 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.054861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.114328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.114286 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8"] Nov 03 20:23:10.115180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.115160 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4edd320a-816a-485d-bb86-5faa0b314b2f" containerName="maas-api" Nov 03 20:23:10.115180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.115179 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edd320a-816a-485d-bb86-5faa0b314b2f" containerName="maas-api" Nov 03 20:23:10.115408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.115324 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edd320a-816a-485d-bb86-5faa0b314b2f" containerName="maas-api" Nov 03 20:23:10.120805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.120784 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.138942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.138915 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8"] Nov 03 20:23:10.210093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.205011 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a619f336-serving-cert\"" Nov 03 20:23:10.210093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.207147 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6mx\" (UniqueName: \"kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.210093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.207266 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.210093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.207305 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.210093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.207474 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.230148 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.211106 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.333109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.331734 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.342661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.309355 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.342661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.340360 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.342661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.341931 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.342661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.342150 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6mx\" (UniqueName: \"kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.350698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.343314 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.364960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.364925 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.365609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.365585 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6mx\" (UniqueName: \"kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.367493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.367464 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs\") pod \"maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.438725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.438683 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:10.438936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.438906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.800292 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:10.800265 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8"] Nov 03 20:23:10.803533 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:10.803503 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d6985c_2a86_4a93_b350_093561287091.slice/crio-97cee07346a93acaabb296442ac6a8d91a36eb074ba8ac1309f10e2ab5df4c97 WatchSource:0}: Error finding container 97cee07346a93acaabb296442ac6a8d91a36eb074ba8ac1309f10e2ab5df4c97: Status 404 returned error can't find the container with id 97cee07346a93acaabb296442ac6a8d91a36eb074ba8ac1309f10e2ab5df4c97 Nov 03 20:23:11.075760 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:11.075721 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" event={"ID":"77d6985c-2a86-4a93-b350-093561287091","Type":"ContainerStarted","Data":"97cee07346a93acaabb296442ac6a8d91a36eb074ba8ac1309f10e2ab5df4c97"} Nov 03 20:23:11.904354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:11.904316 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.011195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.011153 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.091979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.091924 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" event={"ID":"77d6985c-2a86-4a93-b350-093561287091","Type":"ContainerStarted","Data":"e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf"} Nov 03 20:23:12.092890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.092864 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:12.137899 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.137822 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" podStartSLOduration=1.839403611 podStartE2EDuration="2.137800754s" totalImagesPullingTime="298.397143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:12.132391258 +0000 UTC m=+1173.599346513" watchObservedRunningTime="2026-11-03 20:23:12.137800754 +0000 UTC m=+1173.604756008" Nov 03 20:23:12.154282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.154237 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.253167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.253065 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65"] Nov 03 20:23:12.257892 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.257870 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.273754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.273722 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b917c2\"" Nov 03 20:23:12.274306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.274286 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b917c2-dockercfg-qmm7t\"" Nov 03 20:23:12.278352 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.278329 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65"] Nov 03 20:23:12.302615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.302568 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm"] Nov 03 20:23:12.315558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.315526 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.438107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.416235 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.447139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.439207 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.447139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.439340 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4tw\" (UniqueName: \"kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw\") pod \"payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.447139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.439448 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8g2\" (UniqueName: \"kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2\") pod \"payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.461663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.461614 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm"] Nov 03 20:23:12.558140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.547192 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.558140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.547324 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.558140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.547395 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4tw\" (UniqueName: \"kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw\") pod \"payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.558140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.547451 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8g2\" (UniqueName: \"kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2\") pod \"payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.558140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.548669 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.558140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.548877 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.587713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.587648 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8g2\" (UniqueName: \"kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2\") pod \"payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:12.588982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.588611 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4tw\" (UniqueName: \"kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw\") pod \"payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.806628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.806599 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:12.876898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:12.876831 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:13.123944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:13.123920 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm"] Nov 03 20:23:13.125684 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:13.125657 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7473c1_6c0e_4644_89dd_3124415127b2.slice/crio-b8b551dcca890649a4c733d3469d590e697a4b43a568b2c879f508b442ae8390 WatchSource:0}: Error finding container b8b551dcca890649a4c733d3469d590e697a4b43a568b2c879f508b442ae8390: Status 404 returned error can't find the container with id b8b551dcca890649a4c733d3469d590e697a4b43a568b2c879f508b442ae8390 Nov 03 20:23:13.212061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:13.212032 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65"] Nov 03 20:23:13.214151 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:13.214123 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddad4d1d_a61a_457e_8523_b4021250089f.slice/crio-6bee8580046cdec1731ef9d63ebb877b1864d7fa582de9a5ed242dd34adc3441 WatchSource:0}: Error finding container 6bee8580046cdec1731ef9d63ebb877b1864d7fa582de9a5ed242dd34adc3441: Status 404 returned error can't find the container with id 6bee8580046cdec1731ef9d63ebb877b1864d7fa582de9a5ed242dd34adc3441 Nov 03 20:23:13.761942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:13.761907 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:14.112024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.111988 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" event={"ID":"ddad4d1d-a61a-457e-8523-b4021250089f","Type":"ContainerStarted","Data":"460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2"} Nov 03 20:23:14.112268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.112051 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" event={"ID":"ddad4d1d-a61a-457e-8523-b4021250089f","Type":"ContainerStarted","Data":"6bee8580046cdec1731ef9d63ebb877b1864d7fa582de9a5ed242dd34adc3441"} Nov 03 20:23:14.112499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.112465 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:14.114238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.114211 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" event={"ID":"1a7473c1-6c0e-4644-89dd-3124415127b2","Type":"ContainerStarted","Data":"2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e"} Nov 03 20:23:14.114338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.114252 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" event={"ID":"1a7473c1-6c0e-4644-89dd-3124415127b2","Type":"ContainerStarted","Data":"b8b551dcca890649a4c733d3469d590e697a4b43a568b2c879f508b442ae8390"} Nov 03 20:23:14.114338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.114294 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:14.150066 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.150019 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" podStartSLOduration=2.150007248 podStartE2EDuration="2.150007248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:14.147583225 +0000 UTC m=+1175.614538478" watchObservedRunningTime="2026-11-03 20:23:14.150007248 +0000 UTC m=+1175.616962502" Nov 03 20:23:14.188985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:14.188924 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" podStartSLOduration=2.188909034 podStartE2EDuration="2.188909034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:14.183414549 +0000 UTC m=+1175.650369834" watchObservedRunningTime="2026-11-03 20:23:14.188909034 +0000 UTC m=+1175.655864288" Nov 03 20:23:15.383812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:15.383778 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.864157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:16.864089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:18.107942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:18.107912 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:18.393845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:18.393760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:19.916792 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:19.916759 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:21.804246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:21.804212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:23.293429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:23.293392 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:24.863396 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:24.863360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:25.128960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:25.128871 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:23:25.129182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:25.128982 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:23:26.632986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.632952 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx"] Nov 03 20:23:26.638296 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.638273 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.657642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.657616 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx"] Nov 03 20:23:26.663998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.663970 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-b917c2-serving-cert\"" Nov 03 20:23:26.722470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.722432 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.726485 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.726449 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.726689 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.726667 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkk6\" (UniqueName: \"kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.726763 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.726707 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.726763 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.726749 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.828501 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.828468 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkk6\" (UniqueName: \"kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.828697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.828506 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.828697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.828548 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.828823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.828797 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.829324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.829298 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.831044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.831004 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.831479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.831459 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.831562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.831493 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.847503 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.847475 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkk6\" (UniqueName: \"kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6\") pod \"maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:26.932565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.932475 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.956322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:26.956287 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:27.287551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:27.287519 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx"] Nov 03 20:23:27.289907 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:27.289866 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4eced3_9286_4b3a_9d75_0fe80735f8b5.slice/crio-852c14077a391de5f27bb66cc5431050886734b80421cb1b5fb9daf5c7be2595 WatchSource:0}: Error finding container 852c14077a391de5f27bb66cc5431050886734b80421cb1b5fb9daf5c7be2595: Status 404 returned error can't find the container with id 852c14077a391de5f27bb66cc5431050886734b80421cb1b5fb9daf5c7be2595 Nov 03 20:23:28.252696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:28.252657 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" event={"ID":"8f4eced3-9286-4b3a-9d75-0fe80735f8b5","Type":"ContainerStarted","Data":"d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5"} Nov 03 20:23:28.253226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:28.252732 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" event={"ID":"8f4eced3-9286-4b3a-9d75-0fe80735f8b5","Type":"ContainerStarted","Data":"852c14077a391de5f27bb66cc5431050886734b80421cb1b5fb9daf5c7be2595"} Nov 03 20:23:28.253226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:28.252779 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:28.297565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:28.297496 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" podStartSLOduration=1.933913963 podStartE2EDuration="2.297473091s" totalImagesPullingTime="363.559128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:28.290456396 +0000 UTC m=+1189.757411651" watchObservedRunningTime="2026-11-03 20:23:28.297473091 +0000 UTC m=+1189.764428348" Nov 03 20:23:28.627440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:28.627388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:29.704200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:29.704155 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:30.515476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:30.515254 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:30.635766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:30.635726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.408400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:32.408365 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:34.202496 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:34.202460 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:34.277147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:34.277114 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:23:35.636347 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:23:35.636293 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:23:35.760036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:35.760001 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:35.892496 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:35.892402 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:36.005136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:36.003727 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:37.260217 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.260184 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm"] Nov 03 20:23:37.260700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.260422 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" podUID="1070ca08-7ee4-4883-bb63-c8fd043226ae" containerName="maas-api" containerID="cri-o://d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655" gracePeriod=30 Nov 03 20:23:37.315258 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.315220 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:37.754100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.732465 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:37.879100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.878864 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:37.907224 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.906291 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs\") pod \"1070ca08-7ee4-4883-bb63-c8fd043226ae\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " Nov 03 20:23:37.907224 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.906534 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls\") pod \"1070ca08-7ee4-4883-bb63-c8fd043226ae\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " Nov 03 20:23:37.907224 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.906717 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj\" (UniqueName: \"kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj\") pod \"1070ca08-7ee4-4883-bb63-c8fd043226ae\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " Nov 03 20:23:37.907224 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.906748 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca\") pod \"1070ca08-7ee4-4883-bb63-c8fd043226ae\" (UID: \"1070ca08-7ee4-4883-bb63-c8fd043226ae\") " Nov 03 20:23:37.910417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.907666 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca" pod "1070ca08-7ee4-4883-bb63-c8fd043226ae" (UID: "1070ca08-7ee4-4883-bb63-c8fd043226ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:37.910417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.909987 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs" pod "1070ca08-7ee4-4883-bb63-c8fd043226ae" (UID: "1070ca08-7ee4-4883-bb63-c8fd043226ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:37.910417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.910135 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj" pod "1070ca08-7ee4-4883-bb63-c8fd043226ae" (UID: "1070ca08-7ee4-4883-bb63-c8fd043226ae"). InnerVolumeSpecName "kube-api-access-tgvpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:37.910417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:37.910342 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls" pod "1070ca08-7ee4-4883-bb63-c8fd043226ae" (UID: "1070ca08-7ee4-4883-bb63-c8fd043226ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:38.019866 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.018301 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:38.020155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.020131 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1070ca08-7ee4-4883-bb63-c8fd043226ae-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:38.020291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.020276 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvpj\" (UniqueName: \"kubernetes.io/projected/1070ca08-7ee4-4883-bb63-c8fd043226ae-kube-api-access-tgvpj\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:38.020398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.020383 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1070ca08-7ee4-4883-bb63-c8fd043226ae-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:38.035803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.035493 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.197182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.192576 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.350778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.350745 5043 generic.go:356] "Generic (PLEG): container finished" podID="1070ca08-7ee4-4883-bb63-c8fd043226ae" containerID="d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655" exitCode=0 Nov 03 20:23:38.351717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.350805 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" Nov 03 20:23:38.351717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.350828 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" event={"ID":"1070ca08-7ee4-4883-bb63-c8fd043226ae","Type":"ContainerDied","Data":"d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655"} Nov 03 20:23:38.351717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.350883 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm" event={"ID":"1070ca08-7ee4-4883-bb63-c8fd043226ae","Type":"ContainerDied","Data":"57dbd59ac7d291862761af698d99ad9f08e302256759c640998b4eb1714f98a9"} Nov 03 20:23:38.351717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.350911 5043 scope.go:122] "RemoveContainer" containerID="d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655" Nov 03 20:23:38.365715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.365689 5043 scope.go:122] "RemoveContainer" containerID="d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655" Nov 03 20:23:38.366048 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:23:38.366023 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655\": container with ID starting with d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655 not found: ID does not exist" containerID="d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655" Nov 03 20:23:38.366179 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.366061 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655"} err="failed to get container status \"d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655\": rpc error: code = NotFound desc = could not find container \"d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655\": container with ID starting with d0c85cdd50df0adca4b005979cc0b72259053a6c4ac3c60847ce7896a5e10655 not found: ID does not exist" Nov 03 20:23:38.404603 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.404571 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm"] Nov 03 20:23:38.414622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.414593 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-04bf5fc9-d46774fd6-prnkm"] Nov 03 20:23:38.680977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.680938 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1070ca08-7ee4-4883-bb63-c8fd043226ae" path="/var/lib/kubelet/pods/1070ca08-7ee4-4883-bb63-c8fd043226ae/volumes" Nov 03 20:23:38.861155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.861125 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:23:38.868063 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.868036 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:23:38.873127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.873107 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:23:38.874909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:38.874892 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:23:39.603693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:39.603653 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:39.854151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:39.845656 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:40.324136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:40.324058 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:40.777560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:40.777467 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:40.886922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:40.886859 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:41.634467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:41.634431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.299193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:43.299154 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.512512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:43.512222 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.607009 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:43.606974 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:44.345348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:44.345310 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:45.531913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:45.531869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:46.125566 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.125526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:46.328060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.328021 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs"] Nov 03 20:23:46.329306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.329285 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1070ca08-7ee4-4883-bb63-c8fd043226ae" containerName="maas-api" Nov 03 20:23:46.329370 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.329310 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1070ca08-7ee4-4883-bb63-c8fd043226ae" containerName="maas-api" Nov 03 20:23:46.329572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.329561 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1070ca08-7ee4-4883-bb63-c8fd043226ae" containerName="maas-api" Nov 03 20:23:46.335940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.335914 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.353194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.349136 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:46.355403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.355375 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs"] Nov 03 20:23:46.358771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.357429 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-de82c8\"" Nov 03 20:23:46.365149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.365116 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-de82c8-dockercfg-qxt7k\"" Nov 03 20:23:46.508653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.508546 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87n4\" (UniqueName: \"kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4\") pod \"payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.512544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.509426 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.614716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.614671 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87n4\" (UniqueName: \"kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4\") pod \"payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.615203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.614927 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.615693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.615676 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.633294 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.633264 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87n4\" (UniqueName: \"kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4\") pod \"payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.660484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.660451 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:46.704651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.704615 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8"] Nov 03 20:23:46.704884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.704855 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" podUID="77d6985c-2a86-4a93-b350-093561287091" containerName="maas-api" containerID="cri-o://e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf" gracePeriod=30 Nov 03 20:23:46.776405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:46.776373 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.046143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.045821 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs"] Nov 03 20:23:47.331687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.331658 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:47.415964 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.415916 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.474048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.472837 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx\" (UniqueName: \"kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx\") pod \"77d6985c-2a86-4a93-b350-093561287091\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " Nov 03 20:23:47.474048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.473031 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca\") pod \"77d6985c-2a86-4a93-b350-093561287091\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " Nov 03 20:23:47.474048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.473055 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls\") pod \"77d6985c-2a86-4a93-b350-093561287091\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " Nov 03 20:23:47.474048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.473265 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs\") pod \"77d6985c-2a86-4a93-b350-093561287091\" (UID: \"77d6985c-2a86-4a93-b350-093561287091\") " Nov 03 20:23:47.474048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.473575 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca" pod "77d6985c-2a86-4a93-b350-093561287091" (UID: "77d6985c-2a86-4a93-b350-093561287091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:47.474048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.474011 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d6985c-2a86-4a93-b350-093561287091-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:47.477447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.477421 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls" pod "77d6985c-2a86-4a93-b350-093561287091" (UID: "77d6985c-2a86-4a93-b350-093561287091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:47.477700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.477481 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx" pod "77d6985c-2a86-4a93-b350-093561287091" (UID: "77d6985c-2a86-4a93-b350-093561287091"). InnerVolumeSpecName "kube-api-access-5b6mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:47.478755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.478696 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs" pod "77d6985c-2a86-4a93-b350-093561287091" (UID: "77d6985c-2a86-4a93-b350-093561287091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:47.515738 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.515696 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" event={"ID":"098e1578-d007-4299-82cf-d9e05369f350","Type":"ContainerStarted","Data":"0b0686c763816349132532e31a7005c8d6c2459b04a21634eb061988dc37720c"} Nov 03 20:23:47.515929 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.515766 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" event={"ID":"098e1578-d007-4299-82cf-d9e05369f350","Type":"ContainerStarted","Data":"187172c0c7249cf963fae30fca344a8c0520114310c2a8e442667b7b28afaba2"} Nov 03 20:23:47.516049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.516026 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:47.517382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.517356 5043 generic.go:356] "Generic (PLEG): container finished" podID="77d6985c-2a86-4a93-b350-093561287091" containerID="e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf" exitCode=0 Nov 03 20:23:47.517488 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.517407 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" Nov 03 20:23:47.517537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.517407 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" event={"ID":"77d6985c-2a86-4a93-b350-093561287091","Type":"ContainerDied","Data":"e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf"} Nov 03 20:23:47.517537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.517524 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8" event={"ID":"77d6985c-2a86-4a93-b350-093561287091","Type":"ContainerDied","Data":"97cee07346a93acaabb296442ac6a8d91a36eb074ba8ac1309f10e2ab5df4c97"} Nov 03 20:23:47.517610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.517554 5043 scope.go:122] "RemoveContainer" containerID="e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf" Nov 03 20:23:47.529021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.528999 5043 scope.go:122] "RemoveContainer" containerID="e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf" Nov 03 20:23:47.529366 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:23:47.529336 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf\": container with ID starting with e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf not found: ID does not exist" containerID="e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf" Nov 03 20:23:47.529466 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.529371 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf"} err="failed to get container status \"e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf\": rpc error: code = NotFound desc = could not find container \"e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf\": container with ID starting with e2e23dddc64f503edf47bab0113acb72d92be72921652378080f59e02cf22fbf not found: ID does not exist" Nov 03 20:23:47.538955 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.538933 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.556447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.556405 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" podStartSLOduration=1.556390054 podStartE2EDuration="1.556390054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:47.552719623 +0000 UTC m=+1209.019674896" watchObservedRunningTime="2026-11-03 20:23:47.556390054 +0000 UTC m=+1209.023345308" Nov 03 20:23:47.577085 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.577047 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:47.577274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.577092 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d6985c-2a86-4a93-b350-093561287091-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:47.577274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.577110 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6mx\" (UniqueName: \"kubernetes.io/projected/77d6985c-2a86-4a93-b350-093561287091-kube-api-access-5b6mx\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:23:47.593140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.593094 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8"] Nov 03 20:23:47.605897 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.605860 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a619f336-bdf5486c5-vxrn8"] Nov 03 20:23:47.824192 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.823666 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.927807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:47.919346 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:48.563705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:48.563556 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:48.685431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:48.685396 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d6985c-2a86-4a93-b350-093561287091" path="/var/lib/kubelet/pods/77d6985c-2a86-4a93-b350-093561287091/volumes" Nov 03 20:23:49.035598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:49.035557 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.141615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:49.141580 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.823395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:49.823358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:50.195618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:50.195562 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:50.619552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:50.619512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.263760 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:51.263718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.602091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:51.597327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.764011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:51.763979 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.941670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:51.941573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:52.547938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:52.547900 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:52.679561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:52.679501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:53.674363 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:53.674326 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.307291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.307260 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.407311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.407268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.421510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.421485 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw"] Nov 03 20:23:54.422298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.422283 5043 cpu_manager.go:414] "containerMap: removing container" podUID="77d6985c-2a86-4a93-b350-093561287091" containerName="maas-api" Nov 03 20:23:54.422298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.422299 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d6985c-2a86-4a93-b350-093561287091" containerName="maas-api" Nov 03 20:23:54.422436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.422424 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d6985c-2a86-4a93-b350-093561287091" containerName="maas-api" Nov 03 20:23:54.427568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.427551 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.445129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.445058 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw"] Nov 03 20:23:54.454117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.451596 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-503c4e-dockercfg-l2z76\"" Nov 03 20:23:54.454117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.451646 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-503c4e\"" Nov 03 20:23:54.487750 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.487718 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6"] Nov 03 20:23:54.493689 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.493654 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.505117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.505031 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6"] Nov 03 20:23:54.513260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.513226 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.513570 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.513525 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hs62\" (UniqueName: \"kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62\") pod \"payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.600125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.599719 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.678101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.667299 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hs62\" (UniqueName: \"kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62\") pod \"payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.678101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.667418 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.678101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.667584 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.678101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.667781 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtpj\" (UniqueName: \"kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj\") pod \"payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.678101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.669179 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.762216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.762004 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.763736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.763710 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hs62\" (UniqueName: \"kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62\") pod \"payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:54.793493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.790385 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtpj\" (UniqueName: \"kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj\") pod \"payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.793493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.790612 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.793493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.791944 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.819887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.815354 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtpj\" (UniqueName: \"kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj\") pod \"payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.819887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.816548 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:54.863555 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:54.858809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:55.044527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.044497 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:55.163157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.162814 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6"] Nov 03 20:23:55.165315 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:55.165290 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea681b3_4632_45b8_80d4_333939e99c24.slice/crio-e846dbdd003f4be796288518409db61bd88bb3c6b15e10a5a4f087129da6a812 WatchSource:0}: Error finding container e846dbdd003f4be796288518409db61bd88bb3c6b15e10a5a4f087129da6a812: Status 404 returned error can't find the container with id e846dbdd003f4be796288518409db61bd88bb3c6b15e10a5a4f087129da6a812 Nov 03 20:23:55.364936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.364912 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw"] Nov 03 20:23:55.367525 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:55.367495 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e4fe77_f913_44d7_bc46_276176b1c65b.slice/crio-e2a98c27688072c3cb3d06bd98b5b5c1a8546437744b926d1e1013f8ca93df6f WatchSource:0}: Error finding container e2a98c27688072c3cb3d06bd98b5b5c1a8546437744b926d1e1013f8ca93df6f: Status 404 returned error can't find the container with id e2a98c27688072c3cb3d06bd98b5b5c1a8546437744b926d1e1013f8ca93df6f Nov 03 20:23:55.621051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.621015 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:55.737046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.736952 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" event={"ID":"5ea681b3-4632-45b8-80d4-333939e99c24","Type":"ContainerStarted","Data":"71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10"} Nov 03 20:23:55.737046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.737020 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:23:55.737046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.737043 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" event={"ID":"5ea681b3-4632-45b8-80d4-333939e99c24","Type":"ContainerStarted","Data":"e846dbdd003f4be796288518409db61bd88bb3c6b15e10a5a4f087129da6a812"} Nov 03 20:23:55.739210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.739178 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" event={"ID":"a2e4fe77-f913-44d7-bc46-276176b1c65b","Type":"ContainerStarted","Data":"daf8e9147736c098f86f7691ef1aaa252a7019b7595b0ef0105ba2589829a341"} Nov 03 20:23:55.739409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.739229 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" event={"ID":"a2e4fe77-f913-44d7-bc46-276176b1c65b","Type":"ContainerStarted","Data":"e2a98c27688072c3cb3d06bd98b5b5c1a8546437744b926d1e1013f8ca93df6f"} Nov 03 20:23:55.739409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.739260 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:23:55.781735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.781679 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" podStartSLOduration=1.781660502 podStartE2EDuration="1.781660502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:55.774466823 +0000 UTC m=+1217.241422078" watchObservedRunningTime="2026-11-03 20:23:55.781660502 +0000 UTC m=+1217.248615756" Nov 03 20:23:55.810818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:55.810749 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" podStartSLOduration=1.81073088 podStartE2EDuration="1.81073088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:55.81044884 +0000 UTC m=+1217.277404095" watchObservedRunningTime="2026-11-03 20:23:55.81073088 +0000 UTC m=+1217.277686139" Nov 03 20:23:56.403703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.403668 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.501353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.501315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.581936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.581895 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v"] Nov 03 20:23:56.586913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.586889 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.601579 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.601553 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-17b3d9\"" Nov 03 20:23:56.645164 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.644366 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4m8\" (UniqueName: \"kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8\") pod \"payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.645494 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.645469 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.663630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.663541 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-17b3d9-dockercfg-m757k\"" Nov 03 20:23:56.750095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.749561 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4m8\" (UniqueName: \"kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8\") pod \"payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.750095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.750020 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.767719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.764469 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.793090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.786155 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84"] Nov 03 20:23:56.793090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.792555 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:56.852096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.851640 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:56.852096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.851860 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67kz\" (UniqueName: \"kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz\") pod \"payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:56.868090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.861686 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4m8\" (UniqueName: \"kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8\") pod \"payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.902532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.902505 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:56.953999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.953970 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67kz\" (UniqueName: \"kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz\") pod \"payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:56.954300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.954217 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:56.954912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.954888 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:56.973122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:56.973089 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67kz\" (UniqueName: \"kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz\") pod \"payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:57.199716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:57.199641 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:57.295067 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:57.295030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:57.945628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:57.945593 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v"] Nov 03 20:23:57.995764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:57.995729 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84"] Nov 03 20:23:58.423208 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.423184 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84"] Nov 03 20:23:58.426294 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:58.426264 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092a749e_6b8c_47b0_a314_5f1347057618.slice/crio-726043e993bcd6bf60ba847474ba714b2d7f6fd47e4e911507b4bb084adedfe2 WatchSource:0}: Error finding container 726043e993bcd6bf60ba847474ba714b2d7f6fd47e4e911507b4bb084adedfe2: Status 404 returned error can't find the container with id 726043e993bcd6bf60ba847474ba714b2d7f6fd47e4e911507b4bb084adedfe2 Nov 03 20:23:58.436267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.436243 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v"] Nov 03 20:23:58.442228 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:23:58.442198 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cbeb0f_924d_4c06_bb3d_bdd0c00999cb.slice/crio-94923ad6aeb0070c0a1f666895a12e0e1d88872d69041fc7154414089ba3da06 WatchSource:0}: Error finding container 94923ad6aeb0070c0a1f666895a12e0e1d88872d69041fc7154414089ba3da06: Status 404 returned error can't find the container with id 94923ad6aeb0070c0a1f666895a12e0e1d88872d69041fc7154414089ba3da06 Nov 03 20:23:58.564692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.564670 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:23:58.788535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.788451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:58.868773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.868737 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" event={"ID":"092a749e-6b8c-47b0-a314-5f1347057618","Type":"ContainerStarted","Data":"6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432"} Nov 03 20:23:58.868966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.868806 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:23:58.868966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.868835 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" event={"ID":"092a749e-6b8c-47b0-a314-5f1347057618","Type":"ContainerStarted","Data":"726043e993bcd6bf60ba847474ba714b2d7f6fd47e4e911507b4bb084adedfe2"} Nov 03 20:23:58.870522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.870485 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" event={"ID":"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb","Type":"ContainerStarted","Data":"1695f1b6f6ded6d0b3bb2e07ac47c74a52fe119a2f2a850c9110ecf938f5bc5f"} Nov 03 20:23:58.870630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.870548 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:23:58.870630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.870572 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" event={"ID":"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb","Type":"ContainerStarted","Data":"94923ad6aeb0070c0a1f666895a12e0e1d88872d69041fc7154414089ba3da06"} Nov 03 20:23:58.907061 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.907000 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" podStartSLOduration=2.9069829929999997 podStartE2EDuration="2.906982993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:58.904334055 +0000 UTC m=+1220.371289310" watchObservedRunningTime="2026-11-03 20:23:58.906982993 +0000 UTC m=+1220.373938247" Nov 03 20:23:58.915969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.915933 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:58.942773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:23:58.942732 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" podStartSLOduration=2.942721896 podStartE2EDuration="2.942721896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:58.939351832 +0000 UTC m=+1220.406307086" watchObservedRunningTime="2026-11-03 20:23:58.942721896 +0000 UTC m=+1220.409677149" Nov 03 20:24:00.402961 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.402930 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8"] Nov 03 20:24:00.408362 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.408342 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.427630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.427606 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8"] Nov 03 20:24:00.433050 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.433025 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-de82c8-serving-cert\"" Nov 03 20:24:00.495098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.495031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.529614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.529583 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.529758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.529669 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.529847 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.529831 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.529906 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.529894 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcgs\" (UniqueName: \"kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.582940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.582910 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.631978 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.631938 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.632174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.632043 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcgs\" (UniqueName: \"kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.632286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.632267 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.632353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.632316 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.633280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.633253 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.634728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.634703 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.634826 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.634788 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.650155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.650134 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcgs\" (UniqueName: \"kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs\") pod \"maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:00.695152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.693301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.728391 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:00.728348 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:01.073880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:01.073856 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8"] Nov 03 20:24:01.076324 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:24:01.076288 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2045fa93_9dec_4d08_9d76_2dc4122e1306.slice/crio-d82945ad2af3770f5c666e3930dc06a76ffd84b52e814995d7eea9d6e6bcc740 WatchSource:0}: Error finding container d82945ad2af3770f5c666e3930dc06a76ffd84b52e814995d7eea9d6e6bcc740: Status 404 returned error can't find the container with id d82945ad2af3770f5c666e3930dc06a76ffd84b52e814995d7eea9d6e6bcc740 Nov 03 20:24:01.907465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:01.907425 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" event={"ID":"2045fa93-9dec-4d08-9d76-2dc4122e1306","Type":"ContainerStarted","Data":"9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c"} Nov 03 20:24:01.907965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:01.907492 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" event={"ID":"2045fa93-9dec-4d08-9d76-2dc4122e1306","Type":"ContainerStarted","Data":"d82945ad2af3770f5c666e3930dc06a76ffd84b52e814995d7eea9d6e6bcc740"} Nov 03 20:24:01.908377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:01.908355 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:01.945885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:01.945842 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" podStartSLOduration=1.659727289 podStartE2EDuration="1.945831192s" totalImagesPullingTime="286.103903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:01.945414137 +0000 UTC m=+1223.412369410" watchObservedRunningTime="2026-11-03 20:24:01.945831192 +0000 UTC m=+1223.412786446" Nov 03 20:24:02.234427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:02.234336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:03.902593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:03.902554 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.712792 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:04.712757 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.501851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:05.501809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.656390 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:24:05.656349 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:24:06.852952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:06.852916 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:24:06.853410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:06.853000 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:24:06.999561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:06.999525 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.583132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.583052 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w"] Nov 03 20:24:08.589222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.589200 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.607585 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.607560 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w"] Nov 03 20:24:08.613568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.613543 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-503c4e-serving-cert\"" Nov 03 20:24:08.638303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.638268 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbd4z\" (UniqueName: \"kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.638530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.638503 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.638752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.638678 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.638752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.638744 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.679524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.679494 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.739837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.739803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.739975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.739863 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.740055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.739997 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbd4z\" (UniqueName: \"kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.740128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.740066 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.740679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.740655 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.742307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.742284 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.742415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.742397 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.766787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.766761 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbd4z\" (UniqueName: \"kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z\") pod \"maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.900730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.900645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.905637 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.905616 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:08.932914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.932889 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:24:08.994929 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:08.994697 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:09.232827 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:09.232798 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w"] Nov 03 20:24:09.234832 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:24:09.234805 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b70903_bf7e_469e_8542_78c3711b65ec.slice/crio-6158ff269d71c7246a4ff1bf876ef361e73fcf6c3a2982896db252b6b40cac47 WatchSource:0}: Error finding container 6158ff269d71c7246a4ff1bf876ef361e73fcf6c3a2982896db252b6b40cac47: Status 404 returned error can't find the container with id 6158ff269d71c7246a4ff1bf876ef361e73fcf6c3a2982896db252b6b40cac47 Nov 03 20:24:09.880219 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:09.880189 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:24:09.880705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:09.880315 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:24:09.988809 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:09.988771 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" event={"ID":"95b70903-bf7e-469e-8542-78c3711b65ec","Type":"ContainerStarted","Data":"132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a"} Nov 03 20:24:09.989018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:09.988833 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" event={"ID":"95b70903-bf7e-469e-8542-78c3711b65ec","Type":"ContainerStarted","Data":"6158ff269d71c7246a4ff1bf876ef361e73fcf6c3a2982896db252b6b40cac47"} Nov 03 20:24:09.989018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:09.988872 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:10.030394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:10.030336 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" podStartSLOduration=1.703944791 podStartE2EDuration="2.03032159s" totalImagesPullingTime="326.376799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:10.025970248 +0000 UTC m=+1231.492925502" watchObservedRunningTime="2026-11-03 20:24:10.03032159 +0000 UTC m=+1231.497276844" Nov 03 20:24:10.409646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:10.409608 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:11.802685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.802632 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9"] Nov 03 20:24:11.807520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.807498 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:11.824204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.822088 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-17b3d9-serving-cert\"" Nov 03 20:24:11.877545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.877508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:11.914725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.914693 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jskb\" (UniqueName: \"kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:11.914898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.914758 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:11.915001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.914975 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:11.915090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.915039 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:11.974026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:11.973995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.016598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.016565 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jskb\" (UniqueName: \"kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.016752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.016613 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.016752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.016680 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.016867 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.016798 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.017483 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.017459 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.019129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.019105 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.019229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.019153 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.034532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.034512 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jskb\" (UniqueName: \"kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb\") pod \"maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.081169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.078062 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.128129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.128062 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:12.425993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.425924 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9"] Nov 03 20:24:12.861358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:12.861328 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9"] Nov 03 20:24:13.015238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:13.015197 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" event={"ID":"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd","Type":"ContainerStarted","Data":"339ff3d61b31d009d87bcacac860a2475f1d001270b7fa57d20af866e2b29a5c"} Nov 03 20:24:13.593065 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:13.593026 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:14.074757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:14.074716 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" event={"ID":"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd","Type":"ContainerStarted","Data":"d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e"} Nov 03 20:24:14.076353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:14.076323 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:14.121134 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:14.121016 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" podStartSLOduration=2.7104896910000003 podStartE2EDuration="3.120984871s" totalImagesPullingTime="410.49518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:14.114575014 +0000 UTC m=+1235.581530269" watchObservedRunningTime="2026-11-03 20:24:14.120984871 +0000 UTC m=+1235.587940127" Nov 03 20:24:15.106833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:15.106798 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:16.002418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:16.002391 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:24:16.707386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:16.707331 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:18.302490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:18.302454 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:19.909740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:19.909705 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:20.101519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:20.101484 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:24:21.708930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:21.708884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:23.387719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:23.387685 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:24.894731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:24.894698 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:25.834693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:25.834654 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:26.267774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:26.267693 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:26.406838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:26.406799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:26.790040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:26.790003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:27.213171 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:27.213135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.011641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:28.011605 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:29.620260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:29.620222 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:29.843169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:29.843111 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:30.747748 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:30.747706 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.040576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:31.040495 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.166877 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:31.166840 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.450595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:31.450556 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:32.699971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:32.699937 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:32.800626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:32.800591 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:32.986267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:32.986176 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:33.387201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:33.387164 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:33.490725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:33.490689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:33.885514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:33.885477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:34.171868 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:34.171770 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:35.023960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:35.023873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:35.212359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:35.212316 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:35.630485 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:24:35.630444 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:24:36.882126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:36.881671 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:36.988979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:36.988906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:37.336470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:37.336430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:37.462641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:37.462606 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:38.081202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:38.081153 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:38.201556 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:38.201525 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:38.764832 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:38.764788 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:39.727521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:39.727457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:39.840317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:39.840268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:40.627752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:40.627714 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:41.270562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:41.270528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:41.374233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:41.374189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:42.319644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:42.319585 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:42.997632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:42.997596 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:43.096467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:43.096434 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:44.211032 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:44.210995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:45.717666 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:45.717626 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:46.737229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.737193 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xdvwj-mrdtk"] Nov 03 20:24:46.744686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.744662 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.760217 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.760187 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xdvwj-mrdtk"] Nov 03 20:24:46.764806 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.764776 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-95qsx\"" Nov 03 20:24:46.888558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.888523 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcvf\" (UniqueName: \"kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.889280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.888641 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.889280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.888698 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.889280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.888842 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.990099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.989998 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.990260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.990149 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcvf\" (UniqueName: \"kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.990260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.990234 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.990371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.990292 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.990461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.990408 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.991037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.991010 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:46.991426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:46.991402 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:47.023338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.023251 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcvf\" (UniqueName: \"kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf\") pod \"periodic-gathering-xdvwj-mrdtk\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:47.059535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.059505 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:24:47.248354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.248261 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:47.386190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.386145 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xdvwj-mrdtk"] Nov 03 20:24:47.388804 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:24:47.388767 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378af671_3cbd_4532_9baf_f7dd0d4c8e03.slice/crio-1cf6cc18dc0408452d7d20689d7b79da1d81b4632385781220e59ac2ba567bc7 WatchSource:0}: Error finding container 1cf6cc18dc0408452d7d20689d7b79da1d81b4632385781220e59ac2ba567bc7: Status 404 returned error can't find the container with id 1cf6cc18dc0408452d7d20689d7b79da1d81b4632385781220e59ac2ba567bc7 Nov 03 20:24:47.582871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.582832 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" event={"ID":"378af671-3cbd-4532-9baf-f7dd0d4c8e03","Type":"ContainerStarted","Data":"9c1726b86551ded074fc40cceb2a6e98dc05f76b570046b07f65a4ead3adc756"} Nov 03 20:24:47.583035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.582894 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" event={"ID":"378af671-3cbd-4532-9baf-f7dd0d4c8e03","Type":"ContainerStarted","Data":"1cf6cc18dc0408452d7d20689d7b79da1d81b4632385781220e59ac2ba567bc7"} Nov 03 20:24:47.623780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:47.623725 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" podStartSLOduration=1.6237119949999999 podStartE2EDuration="1.623711995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:47.618634735 +0000 UTC m=+1269.085590006" watchObservedRunningTime="2026-11-03 20:24:47.623711995 +0000 UTC m=+1269.090667249" Nov 03 20:24:48.809925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:48.809889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:50.388407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:50.388370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:51.862945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.862911 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld"] Nov 03 20:24:51.868855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.868832 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:51.891318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.891288 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld"] Nov 03 20:24:51.893274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.893249 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-e3ba87-serving-cert\"" Nov 03 20:24:51.943923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.943883 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:51.983898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.983866 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:51.984063 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.984044 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:51.984270 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.984252 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2fj\" (UniqueName: \"kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:51.984343 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:51.984319 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.040898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.040861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:52.105665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.105621 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.111913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.107437 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.111913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.107653 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2fj\" (UniqueName: \"kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.111913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.107744 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.113705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.113621 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.115407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.115238 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.119129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.119053 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.139813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.139084 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2fj\" (UniqueName: \"kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj\") pod \"maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.161627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.161586 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:52.184515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.184146 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:52.529626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.529598 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld"] Nov 03 20:24:52.532292 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:24:52.532267 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10969fd1_504d_4fc7_8236_73fde6043259.slice/crio-fc6291ccf805e751d21a823b270a18cd997fe2a42d196fb095e01e3b5fb79ee0 WatchSource:0}: Error finding container fc6291ccf805e751d21a823b270a18cd997fe2a42d196fb095e01e3b5fb79ee0: Status 404 returned error can't find the container with id fc6291ccf805e751d21a823b270a18cd997fe2a42d196fb095e01e3b5fb79ee0 Nov 03 20:24:52.649571 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:52.649529 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" event={"ID":"10969fd1-504d-4fc7-8236-73fde6043259","Type":"ContainerStarted","Data":"fc6291ccf805e751d21a823b270a18cd997fe2a42d196fb095e01e3b5fb79ee0"} Nov 03 20:24:53.601738 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:53.601697 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:53.667141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:53.667067 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" event={"ID":"10969fd1-504d-4fc7-8236-73fde6043259","Type":"ContainerStarted","Data":"a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9"} Nov 03 20:24:53.668408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:53.668379 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:24:53.709925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:53.709860 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" podStartSLOduration=2.37585816 podStartE2EDuration="2.709840383s" totalImagesPullingTime="333.982223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:53.7060463 +0000 UTC m=+1275.173001571" watchObservedRunningTime="2026-11-03 20:24:53.709840383 +0000 UTC m=+1275.176795639" Nov 03 20:24:55.303936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:55.303897 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:56.923916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:56.923873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:58.774777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:58.774744 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w"] Nov 03 20:24:58.865295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:58.848520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:59.060015 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.059914 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:59.081676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.081640 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w"] Nov 03 20:24:59.081929 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.081779 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.142311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.142280 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a0866c-serving-cert\"" Nov 03 20:24:59.238552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.238491 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w8j4\" (UniqueName: \"kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.238865 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.238838 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.239054 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.239037 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.245344 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.245319 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.347658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.347624 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w8j4\" (UniqueName: \"kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.347856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.347686 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.347856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.347740 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.347953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.347858 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.348154 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:24:59.348130 5043 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-serving-cert: secret "maas-api-e2e-shared-b-w4-a0866c-serving-cert" not found Nov 03 20:24:59.348387 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:24:59.348369 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls podName:8971f511-2a1e-43d9-86bc-2b77098d9982 nodeName:}" failed. No retries permitted until 2026-11-03 20:24:59.84834671 +0000 UTC m=+1281.315301957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls") pod "maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" (UID: "8971f511-2a1e-43d9-86bc-2b77098d9982") : secret "maas-api-e2e-shared-b-w4-a0866c-serving-cert" not found Nov 03 20:24:59.348785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.348759 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.350901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.350879 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.369652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.369626 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w8j4\" (UniqueName: \"kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.494941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.494891 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:59.856479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.856444 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:24:59.861983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:24:59.861957 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:25:00.053444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:00.053403 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:25:00.412573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:00.412509 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w"] Nov 03 20:25:00.642350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:00.641127 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:00.721886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:00.721814 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:25:00.824809 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:00.824768 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" event={"ID":"8971f511-2a1e-43d9-86bc-2b77098d9982","Type":"ContainerStarted","Data":"d300cd55d98772c7d74f200c84f2b36621671a600cfe00f938d40d5be1d28e22"} Nov 03 20:25:01.853975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:01.849892 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" event={"ID":"8971f511-2a1e-43d9-86bc-2b77098d9982","Type":"ContainerStarted","Data":"caee8d91e1db875b5a5d0b8b2c366a36774fe290dde10462333876d763c2a6d5"} Nov 03 20:25:01.853975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:01.851195 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:25:01.905127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:01.892713 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" podStartSLOduration=3.584185111 podStartE2EDuration="3.892692653s" totalImagesPullingTime="308.507542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:01.887523527 +0000 UTC m=+1283.354478783" watchObservedRunningTime="2026-11-03 20:25:01.892692653 +0000 UTC m=+1283.359647908" Nov 03 20:25:02.413963 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:02.413924 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:04.153916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:04.153880 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:04.249696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:04.249659 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:04.352102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:04.352019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:05.215790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:05.215717 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:05.633020 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:25:05.632984 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:25:05.934271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:05.934178 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:07.553905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:07.553865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:08.875446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:08.875415 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:25:09.051834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:09.051804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:10.737469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:10.737432 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:11.390230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:11.390200 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:12.318511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:12.318475 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:13.911129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:13.911048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:15.601620 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:15.601585 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:17.310774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:17.310735 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:19.134710 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:19.134672 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:21.213530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:21.213323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:21.814897 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:21.814855 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:22.923758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:22.923718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:24.591245 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:24.591207 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:26.112095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:26.111890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:27.785458 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:27.785417 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:28.804013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:28.803974 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:29.089290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:29.089250 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:29.407213 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:29.407113 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:29.819977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:29.817522 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:30.071174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:30.070657 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:30.916815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:30.916535 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:31.036613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:31.036570 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.120800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:32.120759 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.840975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:32.840942 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:34.076667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:34.076571 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:34.336648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:34.336559 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:34.462095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:34.458926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:35.300515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:35.300477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:35.679289 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:25:35.679225 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:25:36.321886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:36.321851 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:36.793995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:36.793956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:36.940036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:36.939903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:37.248093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:37.237804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:38.108703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:38.108661 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:38.213576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:38.213540 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:39.022790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:39.022753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:40.720998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:40.720964 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:41.714297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:41.714259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:42.332102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:42.329382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:42.651484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:42.651396 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:42.752318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:42.752275 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:44.390413 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:44.390380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:45.832731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:45.832695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.052575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:46.052540 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.200837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:46.200758 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.750714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:46.750664 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.857276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:46.857238 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:47.907207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:47.907175 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:48.014699 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:48.014662 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:48.119044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:48.119014 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:49.819495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:49.819458 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:50.380216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:50.380170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:50.476605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:50.476564 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:51.695916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:51.695881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:52.559156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.559118 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx"] Nov 03 20:25:52.565427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.565404 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.591922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.591880 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx"] Nov 03 20:25:52.674299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.674259 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eb87f429-openshift-default-dockercfg-fwjlv\"" Nov 03 20:25:52.681236 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.681195 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.687842 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.687807 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxk6d\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.688206 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.688179 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.689675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.688498 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.689925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.689907 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.690149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.690130 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.690803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.690774 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.691017 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.690970 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.691314 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.691293 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.793531 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793493 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793603 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793732 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793820 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793877 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxk6d\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793919 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793994 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.793993 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.794030 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.794095 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.794496 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.794502 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.794818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.794775 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.795231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.795205 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.796974 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.796954 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.797375 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.797353 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.812693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.812628 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxk6d\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.813303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.813278 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:52.883226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:52.883188 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:53.213393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.213358 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx"] Nov 03 20:25:53.216020 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:25:53.215988 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ae8950_3221_420c_b2eb_fd4035002a6f.slice/crio-a40101d3ecee96ea29dfd232dda6c47fd3b4ce2d8cbd02467e96e25efeccc4e2 WatchSource:0}: Error finding container a40101d3ecee96ea29dfd232dda6c47fd3b4ce2d8cbd02467e96e25efeccc4e2: Status 404 returned error can't find the container with id a40101d3ecee96ea29dfd232dda6c47fd3b4ce2d8cbd02467e96e25efeccc4e2 Nov 03 20:25:53.219595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.219554 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:53.219688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.219646 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:53.219733 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.219689 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:53.281981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.281944 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:53.885742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.885702 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" event={"ID":"e3ae8950-3221-420c-b2eb-fd4035002a6f","Type":"ContainerStarted","Data":"2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e"} Nov 03 20:25:53.886186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.885770 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" event={"ID":"e3ae8950-3221-420c-b2eb-fd4035002a6f","Type":"ContainerStarted","Data":"a40101d3ecee96ea29dfd232dda6c47fd3b4ce2d8cbd02467e96e25efeccc4e2"} Nov 03 20:25:53.954859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:53.954807 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podStartSLOduration=1.9547941149999999 podStartE2EDuration="1.954794115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:53.93935228 +0000 UTC m=+1335.406307534" watchObservedRunningTime="2026-11-03 20:25:53.954794115 +0000 UTC m=+1335.421749369" Nov 03 20:25:54.785811 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:54.785766 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:54.884118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:54.884065 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:25:54.885680 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:54.885649 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:25:54.885831 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:54.885707 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:25:55.884337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:55.884283 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:25:55.885282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:55.884413 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:25:56.276200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:56.276103 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:56.883894 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:56.883859 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:25:56.884062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:56.883913 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:25:57.064059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.064019 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt"] Nov 03 20:25:57.069510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.069486 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.121635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.121431 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt"] Nov 03 20:25:57.171015 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.170932 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-e3ba87-openshift-default-dockercfg-gblhq\"" Nov 03 20:25:57.215215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215189 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215277 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qsl\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215385 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215498 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215620 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215599 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215660 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215635 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215779 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215766 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215800 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.215936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.215916 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.317951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.317913 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qsl\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318086 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318201 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318283 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318319 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318637 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318698 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318699 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318781 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.318888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318873 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.319026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.318965 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.319120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.319020 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.319689 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.319664 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.320960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.320934 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.321089 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.321057 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.336847 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.336819 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.337036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.337016 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qsl\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.418846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.418814 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:57.752860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.752827 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt"] Nov 03 20:25:57.755457 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:25:57.755431 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eff5c5d_7869_41ea_b7fe_f499411cec5a.slice/crio-c1ac5ad0c25d3ecb6ba5effdf11edeeae59f6251774b28d8e15ed51463184039 WatchSource:0}: Error finding container c1ac5ad0c25d3ecb6ba5effdf11edeeae59f6251774b28d8e15ed51463184039: Status 404 returned error can't find the container with id c1ac5ad0c25d3ecb6ba5effdf11edeeae59f6251774b28d8e15ed51463184039 Nov 03 20:25:57.758381 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.758346 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:57.758472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.758415 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:57.758472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.758444 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:57.820914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.820884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:57.884500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.884460 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:25:57.884700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.884539 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:25:57.922944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.922908 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" event={"ID":"2eff5c5d-7869-41ea-b7fe-f499411cec5a","Type":"ContainerStarted","Data":"674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926"} Nov 03 20:25:57.923148 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.922962 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" event={"ID":"2eff5c5d-7869-41ea-b7fe-f499411cec5a","Type":"ContainerStarted","Data":"c1ac5ad0c25d3ecb6ba5effdf11edeeae59f6251774b28d8e15ed51463184039"} Nov 03 20:25:57.966016 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:57.965938 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podStartSLOduration=0.965917984 podStartE2EDuration="965.917984ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:57.961694292 +0000 UTC m=+1339.428649566" watchObservedRunningTime="2026-11-03 20:25:57.965917984 +0000 UTC m=+1339.432873239" Nov 03 20:25:58.418930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:58.418895 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:25:58.420708 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:58.420686 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:25:58.420803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:58.420730 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:25:58.883965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:58.883931 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:25:58.884148 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:58.883985 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:25:59.349312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:59.349273 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:59.420038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:59.420002 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:25:59.420445 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:59.420058 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:25:59.883590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:59.883558 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:25:59.883759 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:25:59.883609 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:00.420030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.419991 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:00.420229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.420052 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:00.884429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.884387 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:00.884596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.884456 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:00.903856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.903826 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg"] Nov 03 20:26:00.908533 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.908497 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:00.928923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.928899 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-55b09bd8-serving-cert\"" Nov 03 20:26:00.938771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.938738 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg"] Nov 03 20:26:00.987361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.981453 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:00.987361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.981571 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:00.987361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.981734 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6gb\" (UniqueName: \"kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:00.987361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.981803 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:00.992131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:00.991360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:01.081611 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.081579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:01.083159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.083131 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.083340 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.083233 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.083428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.083409 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6gb\" (UniqueName: \"kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.083559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.083539 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.084325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.084290 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.086465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.086442 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.086612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.086595 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.102117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.102091 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6gb\" (UniqueName: \"kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb\") pod \"maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.183527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.183439 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:01.231332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.231301 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:01.419960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.419925 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:01.420142 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.419988 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:01.546660 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.546628 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg"] Nov 03 20:26:01.547756 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:26:01.547726 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7a0e4b_32dd_4b90_8e9c_aaa4bcebe468.slice/crio-2a03e0582df1c6660457ca9143364090c8b225f0051ab56228e248cf61bae104 WatchSource:0}: Error finding container 2a03e0582df1c6660457ca9143364090c8b225f0051ab56228e248cf61bae104: Status 404 returned error can't find the container with id 2a03e0582df1c6660457ca9143364090c8b225f0051ab56228e248cf61bae104 Nov 03 20:26:01.883977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.883943 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:01.884118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:01.884001 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:02.023965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.023931 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" event={"ID":"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468","Type":"ContainerStarted","Data":"35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b"} Nov 03 20:26:02.024157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.024008 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" event={"ID":"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468","Type":"ContainerStarted","Data":"2a03e0582df1c6660457ca9143364090c8b225f0051ab56228e248cf61bae104"} Nov 03 20:26:02.024157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.024032 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:02.066155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.062704 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" podStartSLOduration=1.777087469 podStartE2EDuration="2.062686074s" totalImagesPullingTime="285.598605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:02.061617457 +0000 UTC m=+1343.528572729" watchObservedRunningTime="2026-11-03 20:26:02.062686074 +0000 UTC m=+1343.529641331" Nov 03 20:26:02.419664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.419628 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:02.419839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.419691 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:02.625465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.625426 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:02.884523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.884487 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:02.884721 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:02.884549 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:03.419561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:03.419528 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:03.419736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:03.419576 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:03.883526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:03.883491 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:03.883943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:03.883553 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:04.175393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:04.175314 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:04.419811 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:04.419773 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:04.420001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:04.419828 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:04.891004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:04.890958 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:04.891543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:04.891512 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:05.422689 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:05.421198 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:05.422689 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:05.421263 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:05.636056 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:26:05.636008 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:26:05.822536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:05.822457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:05.884504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:05.884457 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:05.884795 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:05.884762 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:06.427470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:06.427432 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:06.428055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:06.428022 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:06.888097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:06.884139 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:06.888097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:06.884197 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:07.420508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:07.420467 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:07.420816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:07.420787 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:07.644056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:07.643999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:07.884486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:07.884456 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:26:07.889230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:07.889197 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:07.890828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:07.890796 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:08.049616 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:08.049583 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:26:08.440136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:08.439749 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:08.440136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:08.439806 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:08.913477 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:08.913439 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:08.914108 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:08.914054 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:09.432094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:09.430721 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:09.432094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:09.430801 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:09.540099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:09.529700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:09.900797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:09.900756 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:09.901007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:09.900847 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:10.419799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:10.419764 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:10.420413 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:10.420382 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:10.886486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:10.886434 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:10.886736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:10.886707 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:11.449998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:11.448534 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:11.449998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:11.448610 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:11.900759 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:11.900725 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:26:11.901130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:11.901047 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:26:11.945589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:11.945125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:12.423893 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:12.423859 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:26:12.424189 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:12.424168 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:26:12.424460 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:12.424157 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 20:26:12.889246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:12.889217 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:26:12.890593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:12.890568 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:26:13.425264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:13.425232 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:26:13.426549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:13.426529 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:26:13.661812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:13.661775 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:13.774199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:13.774101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:13.868457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:13.868424 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:15.369515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:15.369475 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:15.479403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:15.479363 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:15.580178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:15.580140 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:17.046992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:17.046767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:18.632420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:18.632383 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:20.325628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:20.325587 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:21.760635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:21.760600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.410002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:23.409955 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:24.362513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:24.362428 5043 generic.go:356] "Generic (PLEG): container finished" podID="378af671-3cbd-4532-9baf-f7dd0d4c8e03" containerID="9c1726b86551ded074fc40cceb2a6e98dc05f76b570046b07f65a4ead3adc756" exitCode=0 Nov 03 20:26:24.362513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:24.362496 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" event={"ID":"378af671-3cbd-4532-9baf-f7dd0d4c8e03","Type":"ContainerDied","Data":"9c1726b86551ded074fc40cceb2a6e98dc05f76b570046b07f65a4ead3adc756"} Nov 03 20:26:25.104305 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.104264 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:25.644594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.644529 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:26:25.738172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.738141 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path\") pod \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " Nov 03 20:26:25.738351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.738266 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle\") pod \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " Nov 03 20:26:25.738351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.738311 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf\" (UniqueName: \"kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf\") pod \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " Nov 03 20:26:25.738481 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.738441 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle\") pod \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\" (UID: \"378af671-3cbd-4532-9baf-f7dd0d4c8e03\") " Nov 03 20:26:25.738543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.738515 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path" pod "378af671-3cbd-4532-9baf-f7dd0d4c8e03" (UID: "378af671-3cbd-4532-9baf-f7dd0d4c8e03"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:25.738645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.738620 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle" pod "378af671-3cbd-4532-9baf-f7dd0d4c8e03" (UID: "378af671-3cbd-4532-9baf-f7dd0d4c8e03"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:25.739152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.739130 5043 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/378af671-3cbd-4532-9baf-f7dd0d4c8e03-archives-path\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:26:25.739152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.739148 5043 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-service-ca-bundle\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:26:25.739327 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.739152 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle" pod "378af671-3cbd-4532-9baf-f7dd0d4c8e03" (UID: "378af671-3cbd-4532-9baf-f7dd0d4c8e03"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:25.740771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.740747 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf" pod "378af671-3cbd-4532-9baf-f7dd0d4c8e03" (UID: "378af671-3cbd-4532-9baf-f7dd0d4c8e03"). InnerVolumeSpecName "kube-api-access-xqcvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:25.840703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.840674 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqcvf\" (UniqueName: \"kubernetes.io/projected/378af671-3cbd-4532-9baf-f7dd0d4c8e03-kube-api-access-xqcvf\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:26:25.840703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:25.840700 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378af671-3cbd-4532-9baf-f7dd0d4c8e03-trusted-ca-bundle\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:26:26.394897 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:26.394863 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" Nov 03 20:26:26.395403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:26.394893 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xdvwj-mrdtk" event={"ID":"378af671-3cbd-4532-9baf-f7dd0d4c8e03","Type":"ContainerDied","Data":"1cf6cc18dc0408452d7d20689d7b79da1d81b4632385781220e59ac2ba567bc7"} Nov 03 20:26:26.395403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:26.394941 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf6cc18dc0408452d7d20689d7b79da1d81b4632385781220e59ac2ba567bc7" Nov 03 20:26:26.716460 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:26.716386 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:26.982003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:26.981923 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:27.088822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:27.088783 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:28.474156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:28.474099 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:30.116595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:30.116554 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:30.974544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:30.974508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:31.259129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:31.259018 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:31.798139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:31.798110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.445414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:32.445379 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:33.488625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:33.488589 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.127935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:35.127902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.284257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:35.284216 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.420399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:35.420307 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.536022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:35.535984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.635748 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:35.635702 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.637031 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:26:35.636993 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:26:36.137579 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:36.137538 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.250372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:36.250333 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.402409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:36.402317 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.730851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:36.730763 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.915510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:36.915475 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.711499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:38.711461 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.791237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:41.791157 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.019472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:42.019435 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.746492 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:42.746453 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.848943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:42.848894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:43.737104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:43.737044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:45.590587 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:45.590540 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:46.861641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:46.861596 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:47.338005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:47.337976 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:47.480465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:47.480431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.241740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:48.241701 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.402588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:48.400316 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.148090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:49.148042 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:50.572538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:50.572499 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:52.122675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:52.122586 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:52.842523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:52.842485 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.277557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:53.277463 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.731165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:53.729282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.853098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:53.846643 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.260853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:54.260817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.433379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:54.433352 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.533401 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:54.533304 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:55.417987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:55.417946 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.254253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:57.254202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.371704 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:57.371664 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:58.705872 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:26:58.705839 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:00.249694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:00.249657 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:01.706907 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:01.706870 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg"] Nov 03 20:27:01.707596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:01.707545 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" podUID="ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" containerName="maas-api" containerID="cri-o://35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b" gracePeriod=30 Nov 03 20:27:01.763413 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:01.763380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.212818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.212786 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.271183 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc"] Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.272228 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" containerName="maas-api" Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.272249 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" containerName="maas-api" Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.272321 5043 cpu_manager.go:414] "containerMap: removing container" podUID="378af671-3cbd-4532-9baf-f7dd0d4c8e03" containerName="insights-gathering" Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.272331 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378af671-3cbd-4532-9baf-f7dd0d4c8e03" containerName="insights-gathering" Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.272548 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" containerName="maas-api" Nov 03 20:27:02.273222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.272569 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="378af671-3cbd-4532-9baf-f7dd0d4c8e03" containerName="insights-gathering" Nov 03 20:27:02.288525 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.281358 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.300558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.298639 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d1bfb8-openshift-default-dockercfg-5vmlf\"" Nov 03 20:27:02.309113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.309058 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc"] Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.313403 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb\" (UniqueName: \"kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb\") pod \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.313468 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls\") pod \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.313556 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca\") pod \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.313748 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs\") pod \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\" (UID: \"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468\") " Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314098 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314139 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314164 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314207 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314274 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314320 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314384 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314519 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgr7\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314547 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.319157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.314884 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca" pod "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" (UID: "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:02.320210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.320182 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs" pod "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" (UID: "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:02.325427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.325389 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls" pod "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" (UID: "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:27:02.327122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.327058 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb" pod "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" (UID: "ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468"). InnerVolumeSpecName "kube-api-access-df6gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:02.416188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416099 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416141 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416171 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416234 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416284 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416362 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416547 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416568 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgr7\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416623 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416722 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416816 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.416959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416920 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6gb\" (UniqueName: \"kubernetes.io/projected/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-kube-api-access-df6gb\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:27:02.416959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416939 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:27:02.416959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416956 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:27:02.417199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.416971 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:27:02.417257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.417230 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.417345 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.417306 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.417404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.417380 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.419252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.419234 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.419351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.419269 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.435000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.434974 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.435223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.435206 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgr7\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7\") pod \"e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.550093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.548680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.614595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.614545 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:02.764354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.763370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.857165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.839683 5043 generic.go:356] "Generic (PLEG): container finished" podID="ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" containerID="35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b" exitCode=0 Nov 03 20:27:02.857165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.839770 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" event={"ID":"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468","Type":"ContainerDied","Data":"35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b"} Nov 03 20:27:02.857165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.839852 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" event={"ID":"ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468","Type":"ContainerDied","Data":"2a03e0582df1c6660457ca9143364090c8b225f0051ab56228e248cf61bae104"} Nov 03 20:27:02.857165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.839878 5043 scope.go:122] "RemoveContainer" containerID="35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b" Nov 03 20:27:02.857165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.840148 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg" Nov 03 20:27:02.874561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.867892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.880979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.880959 5043 scope.go:122] "RemoveContainer" containerID="35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b" Nov 03 20:27:02.881351 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:27:02.881325 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b\": container with ID starting with 35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b not found: ID does not exist" containerID="35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b" Nov 03 20:27:02.881446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.881365 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b"} err="failed to get container status \"35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b\": rpc error: code = NotFound desc = could not find container \"35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b\": container with ID starting with 35f1dc8d6caf2fd3dd2cb38ada43f778c400ee6350168876e7361db62c5d760b not found: ID does not exist" Nov 03 20:27:02.912455 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.912247 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg"] Nov 03 20:27:02.918421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:02.918298 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55b09bd8-69c587759c-qzxpg"] Nov 03 20:27:03.093044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.092978 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc"] Nov 03 20:27:03.095330 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:27:03.095251 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe65ccf6_55cd_426f_8c1a_b677eddeb02f.slice/crio-f3a3074b2d26ece87c3bb38bde736e9c9fff4c552d4059481702b55060471be5 WatchSource:0}: Error finding container f3a3074b2d26ece87c3bb38bde736e9c9fff4c552d4059481702b55060471be5: Status 404 returned error can't find the container with id f3a3074b2d26ece87c3bb38bde736e9c9fff4c552d4059481702b55060471be5 Nov 03 20:27:03.099132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.099090 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:03.099237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.099187 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:03.099294 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.099253 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:03.196396 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.196364 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.347484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.347445 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.451573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.451482 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.854947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.854909 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" event={"ID":"be65ccf6-55cd-426f-8c1a-b677eddeb02f","Type":"ContainerStarted","Data":"2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade"} Nov 03 20:27:03.855567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.855526 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" event={"ID":"be65ccf6-55cd-426f-8c1a-b677eddeb02f","Type":"ContainerStarted","Data":"f3a3074b2d26ece87c3bb38bde736e9c9fff4c552d4059481702b55060471be5"} Nov 03 20:27:03.921690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:03.921647 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podStartSLOduration=10.921635007 podStartE2EDuration="10.921635007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:03.909996432 +0000 UTC m=+1405.376951686" watchObservedRunningTime="2026-11-03 20:27:03.921635007 +0000 UTC m=+1405.388590261" Nov 03 20:27:04.504345 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:04.504315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:04.615128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:04.615090 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:04.623240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:04.623208 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:04.681156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:04.681118 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468" path="/var/lib/kubelet/pods/ea7a0e4b-32dd-4b90-8e9c-aaa4bcebe468/volumes" Nov 03 20:27:04.864168 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:04.864140 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:04.865145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:04.865124 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:27:05.065426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:05.065385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.220136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:05.219962 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.480155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:05.473789 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.636693 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:27:05.636647 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:27:06.269490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:06.269452 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:06.393112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:06.393063 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:06.498637 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:06.498597 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:06.610092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:06.607218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.099352 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.099313 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.202679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.202645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.240859 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.240827 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5"] Nov 03 20:27:08.248251 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.248230 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.263828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.263803 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5"] Nov 03 20:27:08.267528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.267507 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d1bfb8\"" Nov 03 20:27:08.338098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.330475 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d1bfb8-dockercfg-rdx9w\"" Nov 03 20:27:08.370249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.353265 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.412094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.371909 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.412094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.371985 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5f68\" (UniqueName: \"kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68\") pod \"payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.477101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.476360 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.477101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.476441 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5f68\" (UniqueName: \"kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68\") pod \"payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.508097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.478045 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.612101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.598112 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5f68\" (UniqueName: \"kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68\") pod \"payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:08.867527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:08.867497 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:09.210044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.210018 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5"] Nov 03 20:27:09.214541 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:27:09.214515 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438055de_8964_4c06_a10b_786b96c44d8b.slice/crio-dad638ab2bf953f9a2c9096da0aa2baa39465138df6214a4ce7d51cfde65e2b7 WatchSource:0}: Error finding container dad638ab2bf953f9a2c9096da0aa2baa39465138df6214a4ce7d51cfde65e2b7: Status 404 returned error can't find the container with id dad638ab2bf953f9a2c9096da0aa2baa39465138df6214a4ce7d51cfde65e2b7 Nov 03 20:27:09.633115 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.633021 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.734015 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.733980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.853157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.853124 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.950429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.950341 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" event={"ID":"438055de-8964-4c06-a10b-786b96c44d8b","Type":"ContainerStarted","Data":"931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c"} Nov 03 20:27:09.950606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.950431 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" event={"ID":"438055de-8964-4c06-a10b-786b96c44d8b","Type":"ContainerStarted","Data":"dad638ab2bf953f9a2c9096da0aa2baa39465138df6214a4ce7d51cfde65e2b7"} Nov 03 20:27:09.950606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.950478 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:09.965781 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.965748 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.992035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:09.991985 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" podStartSLOduration=1.991971451 podStartE2EDuration="1.991971451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:09.988937841 +0000 UTC m=+1411.455893096" watchObservedRunningTime="2026-11-03 20:27:09.991971451 +0000 UTC m=+1411.458926704" Nov 03 20:27:10.065903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:10.065873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:10.872003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:10.871967 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:10.971770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:10.971738 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:11.508348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:11.508298 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:13.111062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:13.111017 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:14.785874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:14.784353 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:15.363066 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.363027 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7030-6cbfeb"] Nov 03 20:27:15.369289 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.369258 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:15.382947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.382242 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7030-6cbfeb"] Nov 03 20:27:15.433712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.433685 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2hbv\"" Nov 03 20:27:15.441874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.441847 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cvs\" (UniqueName: \"kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs\") pod \"test-curl-7030-6cbfeb\" (UID: \"c41798d4-fd77-4b7c-8328-4619484b3fde\") " pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:15.542834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.542802 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cvs\" (UniqueName: \"kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs\") pod \"test-curl-7030-6cbfeb\" (UID: \"c41798d4-fd77-4b7c-8328-4619484b3fde\") " pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:15.560731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.560707 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cvs\" (UniqueName: \"kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs\") pod \"test-curl-7030-6cbfeb\" (UID: \"c41798d4-fd77-4b7c-8328-4619484b3fde\") " pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:15.685592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:15.685512 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:16.055962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:16.055938 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7030-6cbfeb"] Nov 03 20:27:16.058241 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:27:16.058213 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc41798d4_fd77_4b7c_8328_4619484b3fde.slice/crio-8c3181da688c3e02d5e315bcd04dc10db6cb27f02291bf66ba8c83201c409c38 WatchSource:0}: Error finding container 8c3181da688c3e02d5e315bcd04dc10db6cb27f02291bf66ba8c83201c409c38: Status 404 returned error can't find the container with id 8c3181da688c3e02d5e315bcd04dc10db6cb27f02291bf66ba8c83201c409c38 Nov 03 20:27:16.060228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:16.060205 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:27:16.400151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:16.400112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:17.045354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:17.045309 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7030-6cbfeb" event={"ID":"c41798d4-fd77-4b7c-8328-4619484b3fde","Type":"ContainerStarted","Data":"8c3181da688c3e02d5e315bcd04dc10db6cb27f02291bf66ba8c83201c409c38"} Nov 03 20:27:17.938389 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:17.938344 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:18.059658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:18.059617 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7030-6cbfeb" event={"ID":"c41798d4-fd77-4b7c-8328-4619484b3fde","Type":"ContainerStarted","Data":"31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73"} Nov 03 20:27:18.095453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:18.095398 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7030-6cbfeb" podStartSLOduration=2.059589472 podStartE2EDuration="3.095384239s" totalImagesPullingTime="1.035794767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:18.094885744 +0000 UTC m=+1419.561840997" watchObservedRunningTime="2026-11-03 20:27:18.095384239 +0000 UTC m=+1419.562339492" Nov 03 20:27:18.786281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:18.786244 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:18.935209 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:18.935180 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7030-6cbfeb"] Nov 03 20:27:18.941228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:18.941197 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7030-6cbfeb"] Nov 03 20:27:19.544730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:19.544695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:20.075863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:20.075819 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7030-6cbfeb" podUID="c41798d4-fd77-4b7c-8328-4619484b3fde" containerName="test-curl-7030-6cbfeb" containerID="cri-o://31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73" gracePeriod=2 Nov 03 20:27:20.088562 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:27:20.088526 5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7030-6cbfeb\" is forbidden: User \"system:node:ip-10-0-140-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-102.ec2.internal' and this object" podUID="c41798d4-fd77-4b7c-8328-4619484b3fde" pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:20.966456 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:20.966428 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:27:21.276643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:21.276557 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:21.633400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:21.633349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:21.917327 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:21.917221 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:22.295448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:22.295424 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:22.366300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:22.366251 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs\" (UniqueName: \"kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs\") pod \"c41798d4-fd77-4b7c-8328-4619484b3fde\" (UID: \"c41798d4-fd77-4b7c-8328-4619484b3fde\") " Nov 03 20:27:22.368630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:22.368597 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs" pod "c41798d4-fd77-4b7c-8328-4619484b3fde" (UID: "c41798d4-fd77-4b7c-8328-4619484b3fde"). InnerVolumeSpecName "kube-api-access-59cvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:22.468706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:22.468672 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59cvs\" (UniqueName: \"kubernetes.io/projected/c41798d4-fd77-4b7c-8328-4619484b3fde-kube-api-access-59cvs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:27:22.681298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:22.681262 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41798d4-fd77-4b7c-8328-4619484b3fde" path="/var/lib/kubelet/pods/c41798d4-fd77-4b7c-8328-4619484b3fde/volumes" Nov 03 20:27:23.101503 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:23.101472 5043 generic.go:356] "Generic (PLEG): container finished" podID="c41798d4-fd77-4b7c-8328-4619484b3fde" containerID="31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73" exitCode=137 Nov 03 20:27:23.101677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:23.101519 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-6cbfeb" Nov 03 20:27:23.101677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:23.101569 5043 scope.go:122] "RemoveContainer" containerID="31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73" Nov 03 20:27:23.114303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:23.113534 5043 scope.go:122] "RemoveContainer" containerID="31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73" Nov 03 20:27:23.114303 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:27:23.113901 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73\": container with ID starting with 31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73 not found: ID does not exist" containerID="31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73" Nov 03 20:27:23.114303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:23.113938 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73"} err="failed to get container status \"31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73\": rpc error: code = NotFound desc = could not find container \"31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73\": container with ID starting with 31ad5faeea7c68e355fc2d6be7f2760dcf7b0426cf5c2ddc3fcc7f05b6d56c73 not found: ID does not exist" Nov 03 20:27:23.218094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:23.214228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.626264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:24.626221 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.745455 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:24.745398 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:26.268852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:26.268817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:27.940260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:27.940210 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:29.504606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:29.504561 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.242866 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:30.242833 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.352757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:30.352708 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.200614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:31.200567 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.432309 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:31.432264 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.527847 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:31.527755 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.630524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:31.630489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:33.069503 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:33.069468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:34.746594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:34.746555 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:35.309562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:35.309522 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:35.631591 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:27:35.631555 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:27:36.375586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:36.372283 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:36.458258 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:36.458216 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:36.715388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:36.715300 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:38.042778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:38.042734 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:39.628376 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:39.628342 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:41.310879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:41.310834 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:42.997588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:42.997545 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:43.098490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:43.098454 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:44.585283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:44.585245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:46.257303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:46.257263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:47.717382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:47.717340 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:49.373358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:49.373327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:49.521996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:49.521949 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:51.124952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:51.124915 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:52.618585 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:52.618544 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:54.124729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:54.124695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:55.697696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:55.697654 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:57.375787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:57.375753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:58.995301 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:27:58.995263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:00.686544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:00.686498 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.809927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:02.809891 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:04.361943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:04.361911 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:04.745698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:04.745613 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:05.659159 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:28:05.659124 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:28:06.524166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:06.524125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:08.533613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:08.533577 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:10.316536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:10.316500 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:12.085475 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.085428 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:12.453095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.452999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:12.565867 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.563429 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:12.844750 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.844717 5043 generic.go:356] "Generic (PLEG): container finished" podID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerID="2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade" exitCode=0 Nov 03 20:28:12.844955 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.844790 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" event={"ID":"be65ccf6-55cd-426f-8c1a-b677eddeb02f","Type":"ContainerDied","Data":"2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade"} Nov 03 20:28:12.845378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.845359 5043 scope.go:122] "RemoveContainer" containerID="2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade" Nov 03 20:28:12.847584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.847547 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:12.847672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.847637 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:12.847727 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:12.847713 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:13.597930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:13.597884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:13.781292 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:13.781252 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:13.901227 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:13.900802 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" event={"ID":"be65ccf6-55cd-426f-8c1a-b677eddeb02f","Type":"ContainerStarted","Data":"3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a"} Nov 03 20:28:14.129121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:14.126314 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.306149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:14.305243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.621090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:14.619621 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:28:14.621090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:14.619767 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:14.621090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:14.619814 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:28:15.615329 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:15.615298 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:15.615511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:15.615355 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:28:15.713097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:15.712264 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:16.393064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:16.392990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:16.490596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:16.485834 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:16.622546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:16.622513 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:16.622873 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:16.622842 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:28:17.409094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:17.408635 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:17.538727 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:17.538695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:17.615702 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:17.615660 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:17.615866 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:17.615778 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:28:17.816904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:17.816819 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:18.615595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:18.615558 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:18.616133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:18.615620 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:28:19.058621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:19.058507 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:19.163250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:19.163206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:19.619357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:19.619319 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:19.619951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:19.619917 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:28:19.741578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:19.741539 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:19.873654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:19.866174 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:28:20.620204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:20.620175 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:28:20.621371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:20.621347 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:28:21.401220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:21.401169 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:22.979617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:22.979572 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:24.615258 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:24.615213 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:26.233478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:26.233443 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:27.767662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:27.767625 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:27.935191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:27.935146 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:29.505147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:29.505108 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:30.002681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:30.002639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:30.112049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:30.112011 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:30.216429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:30.216391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.103537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:31.103494 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.226100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:31.226029 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.332361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:31.332327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.438647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:31.438550 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:32.995665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:32.995627 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:33.086480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:33.086451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:33.656177 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:33.650587 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:33.983794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:33.983696 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:34.611794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:34.611747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.198458 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:35.198421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.289639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:35.289594 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.409046 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:35.409007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.643878 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:28:35.643845 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:28:37.214035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:37.213997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:38.058912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:38.058869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:38.337565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:38.337520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:38.844288 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:38.844125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:39.012133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.012059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:39.046700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.046657 5043 scope.go:122] "RemoveContainer" containerID="a5cc561b1af150d99ffdba3d49bb6dab3642545f49a63a70baf1408ae21c7cc0" Nov 03 20:28:39.083382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.083346 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:28:39.093170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.093142 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:28:39.095160 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.095016 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:28:39.102814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.102793 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:28:39.249580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.249548 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:39.360144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:39.359408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:40.097416 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:40.097378 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:40.210528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:40.210489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:41.239837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:41.239804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:42.339898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:42.339857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:42.429472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:42.429436 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:42.957034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:42.956998 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:43.161121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:43.161046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:43.260699 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:43.260611 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:44.906502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:44.906463 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:46.485751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:46.485715 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.034953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:48.034900 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.139917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:48.139875 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.457091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:48.457044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:49.307613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:49.307578 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:49.783512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:49.783479 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:51.674095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:51.666582 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:53.558592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:53.557951 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:55.408140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:55.408089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:57.048019 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:57.047978 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:57.760658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:57.760607 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:57.891400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:57.891363 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:59.119095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:28:59.116890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:00.477635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:00.477576 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:00.740771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:00.740690 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:02.388995 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:02.388949 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:03.874945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:03.874901 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.444379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:05.444341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.539045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:05.539012 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.646745 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:05.646706 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.693188 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:05.693154 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:29:07.045744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:07.045704 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:07.149713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:07.149677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:07.248402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:07.248365 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:08.964748 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:08.964709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:09.064970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:09.064936 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:09.245316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:09.245232 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:10.715321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:10.715286 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.310756 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:12.310720 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:13.516275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:13.516237 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:13.988030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:13.987993 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:14.132530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:14.124372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:14.255716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:14.255576 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:15.493874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.493834 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx"] Nov 03 20:29:15.494432 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.494100 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" containerID="cri-o://b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec" gracePeriod=30 Nov 03 20:29:15.747149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.745160 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:15.901596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.901556 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp"] Nov 03 20:29:15.901876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.901841 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" containerID="cri-o://b286e151880344ecebc8883a46232608fe9321abf2f188275531cae89e7fe78c" gracePeriod=30 Nov 03 20:29:15.955805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.955762 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm"] Nov 03 20:29:15.956174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.956045 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" containerID="cri-o://e8c179a1cf4560d99b3cb0a2e364e79ff80d7128a1ad04c5bcfd3a434428b77a" gracePeriod=30 Nov 03 20:29:15.987369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.987336 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp"] Nov 03 20:29:15.987612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:15.987584 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" containerID="cri-o://3cbfd9f537eac4ef8a0c329051273ba1845c779033179c6101f9ac0dfe2024ad" gracePeriod=30 Nov 03 20:29:16.022090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:16.021434 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd"] Nov 03 20:29:16.022090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:16.021666 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" containerID="cri-o://6d86f3ed698d9910fb82d07f99de8f844637f80a7dace04767d2d10cb9e584da" gracePeriod=30 Nov 03 20:29:16.028969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:16.028943 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv"] Nov 03 20:29:16.029208 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:16.029182 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" containerID="cri-o://49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221" gracePeriod=30 Nov 03 20:29:17.445665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:17.445629 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.861117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:17.861028 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.921132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:17.921090 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:17.921308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:17.921162 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:18.058461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:18.058417 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:18.244025 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:18.243926 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9"] Nov 03 20:29:18.244220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:18.244190 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" containerID="cri-o://b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038" gracePeriod=30 Nov 03 20:29:19.337903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:19.337861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:19.561473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:19.561428 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:19.658212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:19.658119 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:19.671239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:19.671206 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:19.671368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:19.671265 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:19.922546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:19.922422 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:20.532055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.532024 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:20.532500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.532133 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:20.553154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.553124 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:29:20.553276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.553197 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 20:29:20.801175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.801152 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:29:20.911942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.911911 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.911968 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912044 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912151 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912196 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912380 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912358 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:20.912439 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912388 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912439 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912404 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:20.912553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912437 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912524 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912658 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket\") pod \"4e4514e4-7a25-4480-8d66-b704218b6098\" (UID: \"4e4514e4-7a25-4480-8d66-b704218b6098\") " Nov 03 20:29:20.912822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912730 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:20.912926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.912901 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:20.913147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.913106 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:20.913563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.913540 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:20.913654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.913569 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:20.913654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.913584 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:20.913654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.913601 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4514e4-7a25-4480-8d66-b704218b6098-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:20.913654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.913613 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:20.915353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.915304 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:20.915448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.915364 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:20.915448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.915405 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "kube-api-access-xhnpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:20.915560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:20.915494 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo" pod "4e4514e4-7a25-4480-8d66-b704218b6098" (UID: "4e4514e4-7a25-4480-8d66-b704218b6098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.023147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.016872 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4514e4-7a25-4480-8d66-b704218b6098-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.023147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.016908 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhnpz\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-kube-api-access-xhnpz\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.023147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.016925 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4514e4-7a25-4480-8d66-b704218b6098-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.023147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.016949 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4514e4-7a25-4480-8d66-b704218b6098-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.135366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.135336 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:29:21.135508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.135388 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 20:29:21.147686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.147600 5043 generic.go:356] "Generic (PLEG): container finished" podID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerID="e8c179a1cf4560d99b3cb0a2e364e79ff80d7128a1ad04c5bcfd3a434428b77a" exitCode=0 Nov 03 20:29:21.147686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.147664 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" event={"ID":"0f0c0970-75c7-4ae1-a683-8d85ebc90779","Type":"ContainerDied","Data":"e8c179a1cf4560d99b3cb0a2e364e79ff80d7128a1ad04c5bcfd3a434428b77a"} Nov 03 20:29:21.149623 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.149596 5043 generic.go:356] "Generic (PLEG): container finished" podID="4e4514e4-7a25-4480-8d66-b704218b6098" containerID="b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec" exitCode=0 Nov 03 20:29:21.149831 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.149809 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" event={"ID":"4e4514e4-7a25-4480-8d66-b704218b6098","Type":"ContainerDied","Data":"b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec"} Nov 03 20:29:21.149901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.149866 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" event={"ID":"4e4514e4-7a25-4480-8d66-b704218b6098","Type":"ContainerDied","Data":"75300191aa10457e3dfd63a7aa22b26007e196142028fe64cc8c40401575ca0c"} Nov 03 20:29:21.149962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.149897 5043 scope.go:122] "RemoveContainer" containerID="b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec" Nov 03 20:29:21.150052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.150039 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx" Nov 03 20:29:21.167155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.166869 5043 generic.go:356] "Generic (PLEG): container finished" podID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerID="3cbfd9f537eac4ef8a0c329051273ba1845c779033179c6101f9ac0dfe2024ad" exitCode=0 Nov 03 20:29:21.167155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.166976 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" event={"ID":"2f4d35fe-8d90-429f-9713-315e2b947b80","Type":"ContainerDied","Data":"3cbfd9f537eac4ef8a0c329051273ba1845c779033179c6101f9ac0dfe2024ad"} Nov 03 20:29:21.212706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.211335 5043 scope.go:122] "RemoveContainer" containerID="b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec" Nov 03 20:29:21.212706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.211594 5043 generic.go:356] "Generic (PLEG): container finished" podID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerID="b286e151880344ecebc8883a46232608fe9321abf2f188275531cae89e7fe78c" exitCode=0 Nov 03 20:29:21.212706 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:21.212503 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec\": container with ID starting with b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec not found: ID does not exist" containerID="b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec" Nov 03 20:29:21.212706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.212542 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec"} err="failed to get container status \"b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec\": rpc error: code = NotFound desc = could not find container \"b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec\": container with ID starting with b335c4c3d55baa63d472930b45acfbe188f64f51275978ce72afea8102d8e0ec not found: ID does not exist" Nov 03 20:29:21.217170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.214515 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" event={"ID":"0e1e8fba-a9bc-4e1d-abd9-99110e67a883","Type":"ContainerDied","Data":"b286e151880344ecebc8883a46232608fe9321abf2f188275531cae89e7fe78c"} Nov 03 20:29:21.217170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.215971 5043 generic.go:356] "Generic (PLEG): container finished" podID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerID="6d86f3ed698d9910fb82d07f99de8f844637f80a7dace04767d2d10cb9e584da" exitCode=0 Nov 03 20:29:21.217170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.216055 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" event={"ID":"a18950f9-3b3c-4610-8f82-67f1eff53bb3","Type":"ContainerDied","Data":"6d86f3ed698d9910fb82d07f99de8f844637f80a7dace04767d2d10cb9e584da"} Nov 03 20:29:21.240161 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.240124 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx"] Nov 03 20:29:21.254369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.254303 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-l5gkx"] Nov 03 20:29:21.333733 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.333710 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:29:21.357107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.357056 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:21.438696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.438657 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.438891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.438723 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.438891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.438799 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.439008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.438907 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.439008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.438945 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.439174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.439095 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.439833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.439810 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.440042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.440016 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:21.471247 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.440184 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.471621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.440513 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.471817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.448358 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.479187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.478125 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.480203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.480181 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.480529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.480409 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.481136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.481105 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.481341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.481324 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket\") pod \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\" (UID: \"0e1e8fba-a9bc-4e1d-abd9-99110e67a883\") " Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483050 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483089 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483105 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483120 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483133 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483144 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.483829 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.483155 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.490241 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.489850 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:29:21.491970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.491949 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.498174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.498137 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj" pod "0e1e8fba-a9bc-4e1d-abd9-99110e67a883" (UID: "0e1e8fba-a9bc-4e1d-abd9-99110e67a883"). InnerVolumeSpecName "kube-api-access-ppvlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.510352 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.510330 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.589305 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.589464 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.589594 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.589747 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.590086 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.590837 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.590144 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.591481 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.599754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.591718 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.602444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.601569 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.602444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.601623 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.602444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.601718 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.602918 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603035 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603144 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603211 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603280 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603384 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603416 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert\") pod \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\" (UID: \"0f0c0970-75c7-4ae1-a683-8d85ebc90779\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603482 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.604825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.603530 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert\") pod \"2f4d35fe-8d90-429f-9713-315e2b947b80\" (UID: \"2f4d35fe-8d90-429f-9713-315e2b947b80\") " Nov 03 20:29:21.605451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.605331 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.607175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.606761 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.607175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.606786 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.607175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.606800 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.607175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.606813 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.607175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.606828 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppvlj\" (UniqueName: \"kubernetes.io/projected/0e1e8fba-a9bc-4e1d-abd9-99110e67a883-kube-api-access-ppvlj\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.607464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.607186 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.607464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.607439 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.607568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.607476 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:21.607640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.607620 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.608630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.608039 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.609231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.609207 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:21.620015 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.619980 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.622147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.622084 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.625368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.625238 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.625368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.625337 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.625368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.625345 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx" pod "2f4d35fe-8d90-429f-9713-315e2b947b80" (UID: "2f4d35fe-8d90-429f-9713-315e2b947b80"). InnerVolumeSpecName "kube-api-access-dq7nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.625565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.625452 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "kube-api-access-r5zhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.625565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.625537 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.628520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.628495 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token" pod "0f0c0970-75c7-4ae1-a683-8d85ebc90779" (UID: "0f0c0970-75c7-4ae1-a683-8d85ebc90779"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.670692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.670665 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:29:21.675188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.675161 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:29:21.708526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708500 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708527 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708542 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708557 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708568 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708580 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708592 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708606 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708619 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0c0970-75c7-4ae1-a683-8d85ebc90779-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708630 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708641 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4d35fe-8d90-429f-9713-315e2b947b80-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708652 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708664 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7nx\" (UniqueName: \"kubernetes.io/projected/2f4d35fe-8d90-429f-9713-315e2b947b80-kube-api-access-dq7nx\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708679 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zhz\" (UniqueName: \"kubernetes.io/projected/0f0c0970-75c7-4ae1-a683-8d85ebc90779-kube-api-access-r5zhz\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.708717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.708692 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4d35fe-8d90-429f-9713-315e2b947b80-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.809536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809494 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.809752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809552 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.809752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809587 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.809752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809618 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.809752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809706 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.809752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809754 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809790 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809826 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.810038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809856 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.810038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809885 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809930 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.809975 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810058 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.810379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810130 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.810379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810253 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:21.810379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810292 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810407 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.810580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810438 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.810580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810560 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket\") pod \"05af44cb-7d4f-4736-a300-62899d4a4fef\" (UID: \"05af44cb-7d4f-4736-a300-62899d4a4fef\") " Nov 03 20:29:21.810726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810617 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo\") pod \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\" (UID: \"a18950f9-3b3c-4610-8f82-67f1eff53bb3\") " Nov 03 20:29:21.810726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810638 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.810896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.810810 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.811269 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.811625 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.811994 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812037 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812058 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05af44cb-7d4f-4736-a300-62899d4a4fef-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812095 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812108 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812122 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812136 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.812357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812150 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.812946 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.812635 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.813273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.813243 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.814123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.814090 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "kube-api-access-qfpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.818712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.818669 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.819972 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.819937 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "kube-api-access-jg426". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.820298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.820185 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:21.820298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.820265 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.820602 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.820580 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.828420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.828391 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy" pod "05af44cb-7d4f-4736-a300-62899d4a4fef" (UID: "05af44cb-7d4f-4736-a300-62899d4a4fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:21.828643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.828623 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo" pod "a18950f9-3b3c-4610-8f82-67f1eff53bb3" (UID: "a18950f9-3b3c-4610-8f82-67f1eff53bb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:21.913970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.913870 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914227 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914265 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914279 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914293 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914308 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914321 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18950f9-3b3c-4610-8f82-67f1eff53bb3-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914334 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914349 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfpv6\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-kube-api-access-qfpv6\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914361 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg426\" (UniqueName: \"kubernetes.io/projected/05af44cb-7d4f-4736-a300-62899d4a4fef-kube-api-access-jg426\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914375 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05af44cb-7d4f-4736-a300-62899d4a4fef-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:21.914755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:21.914392 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18950f9-3b3c-4610-8f82-67f1eff53bb3-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:22.227451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.227351 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" Nov 03 20:29:22.227451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.227377 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm" event={"ID":"0f0c0970-75c7-4ae1-a683-8d85ebc90779","Type":"ContainerDied","Data":"b6378f5843d61dc1527d4f0c50cb5ab6f0e6499bdeb71cf656a65f82f381eb8b"} Nov 03 20:29:22.227451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.227440 5043 scope.go:122] "RemoveContainer" containerID="e8c179a1cf4560d99b3cb0a2e364e79ff80d7128a1ad04c5bcfd3a434428b77a" Nov 03 20:29:22.230282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.230250 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" Nov 03 20:29:22.230424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.230290 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp" event={"ID":"2f4d35fe-8d90-429f-9713-315e2b947b80","Type":"ContainerDied","Data":"21ae59ecafdebc637de4c9f5a1a689fd849f115b705d87007ba868f00ff24103"} Nov 03 20:29:22.232054 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.232031 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" Nov 03 20:29:22.232225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.232114 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp" event={"ID":"0e1e8fba-a9bc-4e1d-abd9-99110e67a883","Type":"ContainerDied","Data":"62de5b80743a9c7218419c01636e69f7abf65562d110c6a67b743122b66d8914"} Nov 03 20:29:22.233910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.233891 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" Nov 03 20:29:22.234154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.233892 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" event={"ID":"a18950f9-3b3c-4610-8f82-67f1eff53bb3","Type":"ContainerDied","Data":"8d67c3334dcd231a1043bde8ecdb883864b7469cc65cef47bd521491b0091471"} Nov 03 20:29:22.235407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.235380 5043 generic.go:356] "Generic (PLEG): container finished" podID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerID="49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221" exitCode=0 Nov 03 20:29:22.235523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.235428 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" Nov 03 20:29:22.235523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.235457 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" event={"ID":"05af44cb-7d4f-4736-a300-62899d4a4fef","Type":"ContainerDied","Data":"49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221"} Nov 03 20:29:22.235523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.235495 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv" event={"ID":"05af44cb-7d4f-4736-a300-62899d4a4fef","Type":"ContainerDied","Data":"9c8b6b7af78aec77c85ef9669497e0551548c5d341d4bbd441d38a11f4d26d02"} Nov 03 20:29:22.242580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.242461 5043 scope.go:122] "RemoveContainer" containerID="3cbfd9f537eac4ef8a0c329051273ba1845c779033179c6101f9ac0dfe2024ad" Nov 03 20:29:22.256991 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.256972 5043 scope.go:122] "RemoveContainer" containerID="b286e151880344ecebc8883a46232608fe9321abf2f188275531cae89e7fe78c" Nov 03 20:29:22.266910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.266889 5043 scope.go:122] "RemoveContainer" containerID="6d86f3ed698d9910fb82d07f99de8f844637f80a7dace04767d2d10cb9e584da" Nov 03 20:29:22.277850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.277830 5043 scope.go:122] "RemoveContainer" containerID="49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221" Nov 03 20:29:22.287695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.287675 5043 scope.go:122] "RemoveContainer" containerID="49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221" Nov 03 20:29:22.288010 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:22.287976 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221\": container with ID starting with 49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221 not found: ID does not exist" containerID="49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221" Nov 03 20:29:22.288133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.288021 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221"} err="failed to get container status \"49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221\": rpc error: code = NotFound desc = could not find container \"49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221\": container with ID starting with 49946c0d231190f99c98fc074f369fe044803c9078c762fb9a8bac8f0879d221 not found: ID does not exist" Nov 03 20:29:22.323282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.323254 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm"] Nov 03 20:29:22.331526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.331503 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-82vdm"] Nov 03 20:29:22.373471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.373445 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp"] Nov 03 20:29:22.377864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.377845 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-rqlmp"] Nov 03 20:29:22.407511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.407486 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv"] Nov 03 20:29:22.414772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.414749 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-m4wlv"] Nov 03 20:29:22.443143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.443121 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp"] Nov 03 20:29:22.447104 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.447082 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-dqmqp"] Nov 03 20:29:22.479463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.479400 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd"] Nov 03 20:29:22.485536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.485512 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd"] Nov 03 20:29:22.609106 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.609059 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:29:22.609480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.609130 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-t9jrd" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:29:22.680688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.680649 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" path="/var/lib/kubelet/pods/05af44cb-7d4f-4736-a300-62899d4a4fef/volumes" Nov 03 20:29:22.681320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.681297 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" path="/var/lib/kubelet/pods/0e1e8fba-a9bc-4e1d-abd9-99110e67a883/volumes" Nov 03 20:29:22.681874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.681847 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" path="/var/lib/kubelet/pods/0f0c0970-75c7-4ae1-a683-8d85ebc90779/volumes" Nov 03 20:29:22.682916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.682894 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" path="/var/lib/kubelet/pods/2f4d35fe-8d90-429f-9713-315e2b947b80/volumes" Nov 03 20:29:22.683437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.683417 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" path="/var/lib/kubelet/pods/4e4514e4-7a25-4480-8d66-b704218b6098/volumes" Nov 03 20:29:22.683941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:22.683922 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" path="/var/lib/kubelet/pods/a18950f9-3b3c-4610-8f82-67f1eff53bb3/volumes" Nov 03 20:29:23.187133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.187059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:23.527131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.527098 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:23.538810 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.538788 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:29:23.660237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.660195 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.660769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.660331 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.660769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.660437 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.660889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.660863 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.660950 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.660938 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.661151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.661057 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.661243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.661171 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.661609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.661557 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.661609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.661566 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.661882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.661856 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.662206 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.662179 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.662348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.662242 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert\") pod \"4e635786-b323-443a-a72e-4006154bf7a8\" (UID: \"4e635786-b323-443a-a72e-4006154bf7a8\") " Nov 03 20:29:23.663222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.662840 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:23.663222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.663153 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e635786-b323-443a-a72e-4006154bf7a8-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.663222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.663175 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.663222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.663188 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.663222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.663201 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.663787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.663766 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.665497 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.665463 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.667410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.667385 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "kube-api-access-79x85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:23.667515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.667473 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:23.667836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.667805 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token" pod "4e635786-b323-443a-a72e-4006154bf7a8" (UID: "4e635786-b323-443a-a72e-4006154bf7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:23.765845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.765730 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.765845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.765761 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e635786-b323-443a-a72e-4006154bf7a8-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.766055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.765855 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79x85\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-kube-api-access-79x85\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.766055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.765883 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e635786-b323-443a-a72e-4006154bf7a8-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:23.766055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:23.765900 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e635786-b323-443a-a72e-4006154bf7a8-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:24.161654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.161618 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:24.261473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.261439 5043 generic.go:356] "Generic (PLEG): container finished" podID="4e635786-b323-443a-a72e-4006154bf7a8" containerID="b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038" exitCode=0 Nov 03 20:29:24.261668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.261488 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" Nov 03 20:29:24.261668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.261528 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" event={"ID":"4e635786-b323-443a-a72e-4006154bf7a8","Type":"ContainerDied","Data":"b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038"} Nov 03 20:29:24.261668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.261588 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9" event={"ID":"4e635786-b323-443a-a72e-4006154bf7a8","Type":"ContainerDied","Data":"2f0e6a1cea644c3c8772d519b1fcb70e1fb7daf174a09e527e1df69a51f88a4c"} Nov 03 20:29:24.261668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.261621 5043 scope.go:122] "RemoveContainer" containerID="b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038" Nov 03 20:29:24.274814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.274795 5043 scope.go:122] "RemoveContainer" containerID="b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038" Nov 03 20:29:24.275115 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:24.275092 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038\": container with ID starting with b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038 not found: ID does not exist" containerID="b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038" Nov 03 20:29:24.275174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.275125 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038"} err="failed to get container status \"b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038\": rpc error: code = NotFound desc = could not find container \"b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038\": container with ID starting with b292bdf6ac9d90309e8525b239bd682ddba85060741f379a6f2b8b4b90644038 not found: ID does not exist" Nov 03 20:29:24.319889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.319859 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9"] Nov 03 20:29:24.329765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.329741 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-lj4c9"] Nov 03 20:29:24.629839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.629803 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:24.681499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.681465 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e635786-b323-443a-a72e-4006154bf7a8" path="/var/lib/kubelet/pods/4e635786-b323-443a-a72e-4006154bf7a8/volumes" Nov 03 20:29:24.731154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.731110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:24.973897 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:24.973813 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:26.463656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:26.462620 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.121100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:27.118133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.240092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:27.232765 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.326525 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:27.326484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.677483 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:27.677448 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.460310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:28.460268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.575574 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:28.575542 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:29.166539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:29.166498 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:30.133032 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:30.132987 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:30.236235 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:30.236192 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:31.063720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:31.063690 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:31.435416 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:31.435380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:33.035414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:33.035371 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:34.787293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:34.787256 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:34.899748 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:34.899653 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:35.026462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:35.026421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:35.140379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:35.140344 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:35.876093 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:35.862612 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:29:36.973256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:36.973207 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:29:36.973776 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:36.973272 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:29:37.438934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:37.438894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:38.817209 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:38.817172 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb"] Nov 03 20:29:38.817704 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:38.817392 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" podUID="7574c52b-4d0d-477a-b151-0d24fd275e46" containerName="maas-api" containerID="cri-o://a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334" gracePeriod=30 Nov 03 20:29:38.884375 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:38.884274 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.221905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.221875 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:29:39.390796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.390709 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls\") pod \"7574c52b-4d0d-477a-b151-0d24fd275e46\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " Nov 03 20:29:39.391158 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.391129 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs\") pod \"7574c52b-4d0d-477a-b151-0d24fd275e46\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " Nov 03 20:29:39.394689 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.394648 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls" pod "7574c52b-4d0d-477a-b151-0d24fd275e46" (UID: "7574c52b-4d0d-477a-b151-0d24fd275e46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:39.406091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.405486 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs" pod "7574c52b-4d0d-477a-b151-0d24fd275e46" (UID: "7574c52b-4d0d-477a-b151-0d24fd275e46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:39.417238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.415961 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca\") pod \"7574c52b-4d0d-477a-b151-0d24fd275e46\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " Nov 03 20:29:39.417238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.416180 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x\" (UniqueName: \"kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x\") pod \"7574c52b-4d0d-477a-b151-0d24fd275e46\" (UID: \"7574c52b-4d0d-477a-b151-0d24fd275e46\") " Nov 03 20:29:39.417238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.417096 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:39.417238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.417119 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7574c52b-4d0d-477a-b151-0d24fd275e46-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:39.417609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.417489 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca" pod "7574c52b-4d0d-477a-b151-0d24fd275e46" (UID: "7574c52b-4d0d-477a-b151-0d24fd275e46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:39.451687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.451646 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x" pod "7574c52b-4d0d-477a-b151-0d24fd275e46" (UID: "7574c52b-4d0d-477a-b151-0d24fd275e46"). InnerVolumeSpecName "kube-api-access-5sd7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:39.519773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.519714 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7574c52b-4d0d-477a-b151-0d24fd275e46-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:39.519773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.519748 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sd7x\" (UniqueName: \"kubernetes.io/projected/7574c52b-4d0d-477a-b151-0d24fd275e46-kube-api-access-5sd7x\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:39.528093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.527182 5043 generic.go:356] "Generic (PLEG): container finished" podID="7574c52b-4d0d-477a-b151-0d24fd275e46" containerID="a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334" exitCode=0 Nov 03 20:29:39.528093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.527455 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" event={"ID":"7574c52b-4d0d-477a-b151-0d24fd275e46","Type":"ContainerDied","Data":"a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334"} Nov 03 20:29:39.528093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.527518 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" event={"ID":"7574c52b-4d0d-477a-b151-0d24fd275e46","Type":"ContainerDied","Data":"8fb948cfc4effd9af70c1cb64160f9b17fcfe8f91becfcee82de3a182f9a4b01"} Nov 03 20:29:39.528093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.527557 5043 scope.go:122] "RemoveContainer" containerID="a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334" Nov 03 20:29:39.528093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.527747 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb" Nov 03 20:29:39.547362 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.546939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.575197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.575172 5043 scope.go:122] "RemoveContainer" containerID="a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334" Nov 03 20:29:39.575623 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:39.575597 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334\": container with ID starting with a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334 not found: ID does not exist" containerID="a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334" Nov 03 20:29:39.575762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.575632 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334"} err="failed to get container status \"a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334\": rpc error: code = NotFound desc = could not find container \"a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334\": container with ID starting with a2719d068aad46b9a5a8df1c7daeed1bd4e574ca1ad010476b51f42305fe0334 not found: ID does not exist" Nov 03 20:29:39.596928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.596895 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb"] Nov 03 20:29:39.606635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.606607 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6fc748-656bb44fb6-jhrnb"] Nov 03 20:29:39.643356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.643268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.747525 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:39.745778 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:40.133030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:40.132985 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:40.684436 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:40.684394 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7574c52b-4d0d-477a-b151-0d24fd275e46" path="/var/lib/kubelet/pods/7574c52b-4d0d-477a-b151-0d24fd275e46/volumes" Nov 03 20:29:41.541550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:41.541506 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:42.639265 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:42.639224 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:43.428645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:43.428593 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:45.314781 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:45.314742 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.208804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.208761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.312411 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.312369 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.412019 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.411966 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.612217 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.612182 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk"] Nov 03 20:29:46.612898 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.612861 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" containerID="cri-o://900863454ad51169450c0db0ea5111576a0d32ab005f4191385a196990c0680f" gracePeriod=30 Nov 03 20:29:46.619944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.619918 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv"] Nov 03 20:29:46.620315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.620288 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" containerID="cri-o://ba0288d22592b67422da942aa2e98e1cd1112b85251baf07397a2ff56bc759e2" gracePeriod=30 Nov 03 20:29:46.630254 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.627339 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp"] Nov 03 20:29:46.630254 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.627594 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" containerID="cri-o://f6f242bd8673b77927cf1d924bb4f0bfc6e40547e4ea10aef79fc690ca6add05" gracePeriod=30 Nov 03 20:29:46.634642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:46.634613 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:47.210847 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:47.210807 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:47.994149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:47.994112 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:47.994673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:47.994162 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:48.146992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.146959 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:48.147183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.147010 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:48.181122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.181063 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:48.181260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.181129 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:48.829824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.829787 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw"] Nov 03 20:29:48.830641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830625 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830642 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830659 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830665 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830686 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830691 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830701 5043 cpu_manager.go:414] "containerMap: removing container" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830706 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830713 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" Nov 03 20:29:48.830720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830718 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830731 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830735 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830747 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830753 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830768 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7574c52b-4d0d-477a-b151-0d24fd275e46" containerName="maas-api" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830773 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7574c52b-4d0d-477a-b151-0d24fd275e46" containerName="maas-api" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830781 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c41798d4-fd77-4b7c-8328-4619484b3fde" containerName="test-curl-7030-6cbfeb" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830785 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41798d4-fd77-4b7c-8328-4619484b3fde" containerName="test-curl-7030-6cbfeb" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830894 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e635786-b323-443a-a72e-4006154bf7a8" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830902 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18950f9-3b3c-4610-8f82-67f1eff53bb3" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830912 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0c0970-75c7-4ae1-a683-8d85ebc90779" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830919 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="05af44cb-7d4f-4736-a300-62899d4a4fef" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830925 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4d35fe-8d90-429f-9713-315e2b947b80" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830931 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4514e4-7a25-4480-8d66-b704218b6098" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830939 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1e8fba-a9bc-4e1d-abd9-99110e67a883" containerName="istio-proxy" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830946 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7574c52b-4d0d-477a-b151-0d24fd275e46" containerName="maas-api" Nov 03 20:29:48.830987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.830951 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41798d4-fd77-4b7c-8328-4619484b3fde" containerName="test-curl-7030-6cbfeb" Nov 03 20:29:48.836596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.836570 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.858457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.858425 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw"] Nov 03 20:29:48.864105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.861923 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c4a71d-openshift-default-dockercfg-gr5wd\"" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914357 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914437 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914515 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9nf\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914592 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914634 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914692 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914770 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914810 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:48.916341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:48.914946 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.016569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016525 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016599 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016649 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9nf\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016724 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016768 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016807 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.016857 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017263 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017330 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017266 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017368 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017467 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017634 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017552 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.017926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.017899 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.019393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.019367 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.019714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.019696 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.035542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.035522 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.035690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.035669 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9nf\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.094048 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.093984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:49.162159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.162121 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:49.269853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.269830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:49.526850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.526814 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw"] Nov 03 20:29:49.529922 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:29:49.529893 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd7e46d_55f9_4d5e_b19e_b1f93c3f6f97.slice/crio-7e6cd096c74390b1d3027eeda279e5b6293cb7e28bfa8d1d51c106488dc1f49d WatchSource:0}: Error finding container 7e6cd096c74390b1d3027eeda279e5b6293cb7e28bfa8d1d51c106488dc1f49d: Status 404 returned error can't find the container with id 7e6cd096c74390b1d3027eeda279e5b6293cb7e28bfa8d1d51c106488dc1f49d Nov 03 20:29:49.532924 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.532892 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:49.533010 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.532960 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:49.533010 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.532996 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:49.641149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.641110 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" event={"ID":"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97","Type":"ContainerStarted","Data":"a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7"} Nov 03 20:29:49.641277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.641177 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" event={"ID":"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97","Type":"ContainerStarted","Data":"7e6cd096c74390b1d3027eeda279e5b6293cb7e28bfa8d1d51c106488dc1f49d"} Nov 03 20:29:49.687362 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.687297 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podStartSLOduration=1.6872775230000001 podStartE2EDuration="1.687277523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:49.682067654 +0000 UTC m=+1571.149022912" watchObservedRunningTime="2026-11-03 20:29:49.687277523 +0000 UTC m=+1571.154232779" Nov 03 20:29:49.905491 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:49.905449 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:50.164610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:50.164537 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:50.166561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:50.166533 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:50.166712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:50.166586 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:50.342287 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:50.342253 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:50.455614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:50.455511 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:51.155188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.155105 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7030-c72573"] Nov 03 20:29:51.160191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.160166 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:51.165109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.165051 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:51.165488 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.165113 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:51.177499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.177476 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7030-c72573"] Nov 03 20:29:51.238140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.236530 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2hbv\"" Nov 03 20:29:51.238140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.237958 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5z5f\" (UniqueName: \"kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f\") pod \"test-curl-7030-c72573\" (UID: \"13942960-ff6b-47fd-a4d3-c2e4da89fa6d\") " pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:51.239781 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.239738 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:51.350098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.337954 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:51.364000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.358962 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5z5f\" (UniqueName: \"kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f\") pod \"test-curl-7030-c72573\" (UID: \"13942960-ff6b-47fd-a4d3-c2e4da89fa6d\") " pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:51.422135 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.411686 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5z5f\" (UniqueName: \"kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f\") pod \"test-curl-7030-c72573\" (UID: \"13942960-ff6b-47fd-a4d3-c2e4da89fa6d\") " pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:51.475687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.475654 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:51.868612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.868565 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7030-c72573"] Nov 03 20:29:51.881110 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:29:51.877527 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13942960_ff6b_47fd_a4d3_c2e4da89fa6d.slice/crio-92c277380eaf26c10cd23ee25ea9de60288dff3bb3695889cccfda8304df3f48 WatchSource:0}: Error finding container 92c277380eaf26c10cd23ee25ea9de60288dff3bb3695889cccfda8304df3f48: Status 404 returned error can't find the container with id 92c277380eaf26c10cd23ee25ea9de60288dff3bb3695889cccfda8304df3f48 Nov 03 20:29:51.957001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.956965 5043 generic.go:356] "Generic (PLEG): container finished" podID="c0d463b1-7917-4095-9755-ee347e7e2955" containerID="900863454ad51169450c0db0ea5111576a0d32ab005f4191385a196990c0680f" exitCode=0 Nov 03 20:29:51.957149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.957124 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" event={"ID":"c0d463b1-7917-4095-9755-ee347e7e2955","Type":"ContainerDied","Data":"900863454ad51169450c0db0ea5111576a0d32ab005f4191385a196990c0680f"} Nov 03 20:29:51.960203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.960175 5043 generic.go:356] "Generic (PLEG): container finished" podID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerID="ba0288d22592b67422da942aa2e98e1cd1112b85251baf07397a2ff56bc759e2" exitCode=0 Nov 03 20:29:51.960386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.960371 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" event={"ID":"d8775583-9b9d-4f0c-a3b2-66584cf64490","Type":"ContainerDied","Data":"ba0288d22592b67422da942aa2e98e1cd1112b85251baf07397a2ff56bc759e2"} Nov 03 20:29:51.962806 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.962782 5043 generic.go:356] "Generic (PLEG): container finished" podID="70a315a8-1afb-496f-a723-81beae6b1d38" containerID="f6f242bd8673b77927cf1d924bb4f0bfc6e40547e4ea10aef79fc690ca6add05" exitCode=0 Nov 03 20:29:51.962915 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.962818 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" event={"ID":"70a315a8-1afb-496f-a723-81beae6b1d38","Type":"ContainerDied","Data":"f6f242bd8673b77927cf1d924bb4f0bfc6e40547e4ea10aef79fc690ca6add05"} Nov 03 20:29:51.964512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:51.964487 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7030-c72573" event={"ID":"13942960-ff6b-47fd-a4d3-c2e4da89fa6d","Type":"ContainerStarted","Data":"92c277380eaf26c10cd23ee25ea9de60288dff3bb3695889cccfda8304df3f48"} Nov 03 20:29:52.053322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.053290 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:29:52.072337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.072285 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.072478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.072359 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.072478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.072484 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.072791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.072544 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.072974 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.072946 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.073096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.073045 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.073192 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.073150 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.073350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.073212 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.073350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.073299 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket\") pod \"c0d463b1-7917-4095-9755-ee347e7e2955\" (UID: \"c0d463b1-7917-4095-9755-ee347e7e2955\") " Nov 03 20:29:52.074353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.074326 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.074535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.074516 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.075229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.075200 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.075345 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.075255 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:52.075546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.075525 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.075737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.075714 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.077716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.077693 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:52.098143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.089608 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "kube-api-access-sqx9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:52.146153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.145977 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token" pod "c0d463b1-7917-4095-9755-ee347e7e2955" (UID: "c0d463b1-7917-4095-9755-ee347e7e2955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:52.165044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.164657 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:52.165044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.164709 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:52.175408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.175380 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.175607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.175591 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d463b1-7917-4095-9755-ee347e7e2955-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.175714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.175703 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d463b1-7917-4095-9755-ee347e7e2955-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.175823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.175809 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqx9j\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-kube-api-access-sqx9j\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.175930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.175906 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.176022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.176010 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.176151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.176138 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.176258 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.176245 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d463b1-7917-4095-9755-ee347e7e2955-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.176368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.176356 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d463b1-7917-4095-9755-ee347e7e2955-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.244495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.244463 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:29:52.247336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.247313 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:29:52.277793 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.277768 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.277913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.277800 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.277913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.277822 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.277913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.277855 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.277913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.277891 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.278182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.277946 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.278182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.278028 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.278182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.278128 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.278334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.278311 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.296119 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.278503 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.296119 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.282470 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:52.296119 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.286199 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320437 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320607 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320679 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320710 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320762 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320792 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320835 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320872 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token\") pod \"d8775583-9b9d-4f0c-a3b2-66584cf64490\" (UID: \"d8775583-9b9d-4f0c-a3b2-66584cf64490\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320904 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320948 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.321191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.320995 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket\") pod \"70a315a8-1afb-496f-a723-81beae6b1d38\" (UID: \"70a315a8-1afb-496f-a723-81beae6b1d38\") " Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.321918 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.321945 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a315a8-1afb-496f-a723-81beae6b1d38-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.321964 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.321980 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.321996 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.321991 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.322260 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.322533 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:52.322874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.322757 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.323532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.323497 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.323626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.323604 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "kube-api-access-52dzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:52.323904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.323880 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:52.329706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.329339 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:52.329706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.329475 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.329706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.329576 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:52.329706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.329573 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5" pod "d8775583-9b9d-4f0c-a3b2-66584cf64490" (UID: "d8775583-9b9d-4f0c-a3b2-66584cf64490"). InnerVolumeSpecName "kube-api-access-c5lr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:52.329706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.329674 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:52.331537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.331513 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token" pod "70a315a8-1afb-496f-a723-81beae6b1d38" (UID: "70a315a8-1afb-496f-a723-81beae6b1d38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424676 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dzv\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-kube-api-access-52dzv\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424713 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lr5\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-kube-api-access-c5lr5\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424731 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424744 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424759 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424774 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8775583-9b9d-4f0c-a3b2-66584cf64490-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424786 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8775583-9b9d-4f0c-a3b2-66584cf64490-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424798 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424810 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8775583-9b9d-4f0c-a3b2-66584cf64490-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424823 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a315a8-1afb-496f-a723-81beae6b1d38-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424836 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a315a8-1afb-496f-a723-81beae6b1d38-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424848 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.424895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.424860 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a315a8-1afb-496f-a723-81beae6b1d38-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:52.913415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:52.913380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:53.022051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.021672 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7030-c72573" event={"ID":"13942960-ff6b-47fd-a4d3-c2e4da89fa6d","Type":"ContainerStarted","Data":"d4f1b1431aee4475d4cc588b15f7fe9b798276b8f65e38178e8423f6d3e84d07"} Nov 03 20:29:53.023474 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.023442 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" Nov 03 20:29:53.023474 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.023463 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk" event={"ID":"c0d463b1-7917-4095-9755-ee347e7e2955","Type":"ContainerDied","Data":"cc3f8b5927f29ab460616eb7ec329c642b2d71035a375eec2b0f56a6f4542601"} Nov 03 20:29:53.023691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.023515 5043 scope.go:122] "RemoveContainer" containerID="900863454ad51169450c0db0ea5111576a0d32ab005f4191385a196990c0680f" Nov 03 20:29:53.028066 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.028044 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" Nov 03 20:29:53.028230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.028095 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv" event={"ID":"d8775583-9b9d-4f0c-a3b2-66584cf64490","Type":"ContainerDied","Data":"a33586cdcdbfaaa4567bf20ad05ed20ea92f578ad356ed0b6d9771b6408df242"} Nov 03 20:29:53.029751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.029728 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" event={"ID":"70a315a8-1afb-496f-a723-81beae6b1d38","Type":"ContainerDied","Data":"c20bd6bfc5dbf8d707e5230c49cfee28502eb152f61662b4155f7165676f9a99"} Nov 03 20:29:53.029751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.029738 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp" Nov 03 20:29:53.038632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.038584 5043 scope.go:122] "RemoveContainer" containerID="ba0288d22592b67422da942aa2e98e1cd1112b85251baf07397a2ff56bc759e2" Nov 03 20:29:53.049623 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.049592 5043 scope.go:122] "RemoveContainer" containerID="f6f242bd8673b77927cf1d924bb4f0bfc6e40547e4ea10aef79fc690ca6add05" Nov 03 20:29:53.060198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.060145 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7030-c72573" podStartSLOduration=2.06012976 podStartE2EDuration="2.06012976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:53.05772713 +0000 UTC m=+1574.524682387" watchObservedRunningTime="2026-11-03 20:29:53.06012976 +0000 UTC m=+1574.527085014" Nov 03 20:29:53.124337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.124295 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp"] Nov 03 20:29:53.130812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.130785 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-82qdp"] Nov 03 20:29:53.165772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.164644 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv"] Nov 03 20:29:53.165772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.165111 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:53.165772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.165167 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:53.170818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.170793 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-t2ccv"] Nov 03 20:29:53.222112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.218332 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk"] Nov 03 20:29:53.222645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:53.222619 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-dcgmk"] Nov 03 20:29:54.127790 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.127754 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7030-c72573"] Nov 03 20:29:54.141356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.141322 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7030-c72573"] Nov 03 20:29:54.165581 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.165546 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:54.165882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.165851 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:54.699096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.688740 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" path="/var/lib/kubelet/pods/70a315a8-1afb-496f-a723-81beae6b1d38/volumes" Nov 03 20:29:54.699096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.689464 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" path="/var/lib/kubelet/pods/c0d463b1-7917-4095-9755-ee347e7e2955/volumes" Nov 03 20:29:54.699096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.690090 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" path="/var/lib/kubelet/pods/d8775583-9b9d-4f0c-a3b2-66584cf64490/volumes" Nov 03 20:29:54.731986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.731950 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm"] Nov 03 20:29:54.732951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.732929 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" Nov 03 20:29:54.732951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.732950 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" Nov 03 20:29:54.733152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.732967 5043 cpu_manager.go:414] "containerMap: removing container" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" containerName="test-curl-7030-c72573" Nov 03 20:29:54.733152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.732972 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" containerName="test-curl-7030-c72573" Nov 03 20:29:54.733152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733015 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" Nov 03 20:29:54.733152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733019 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" Nov 03 20:29:54.733152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733038 5043 cpu_manager.go:414] "containerMap: removing container" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" Nov 03 20:29:54.733152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733043 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" Nov 03 20:29:54.733500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733169 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8775583-9b9d-4f0c-a3b2-66584cf64490" containerName="istio-proxy" Nov 03 20:29:54.733500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733183 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a315a8-1afb-496f-a723-81beae6b1d38" containerName="istio-proxy" Nov 03 20:29:54.733500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733191 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d463b1-7917-4095-9755-ee347e7e2955" containerName="istio-proxy" Nov 03 20:29:54.733500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.733197 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" containerName="test-curl-7030-c72573" Nov 03 20:29:54.738747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.738725 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.754988 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.754961 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm"] Nov 03 20:29:54.769920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.769895 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fab871-serving-cert\"" Nov 03 20:29:54.825090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.824231 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:54.874113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.874043 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.874261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.874203 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.874486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.874457 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8kp\" (UniqueName: \"kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.874671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.874639 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.929257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.929221 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:54.976463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.976379 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8kp\" (UniqueName: \"kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.976463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.976436 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.976838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.976513 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.976838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.976567 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:54.976838 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:54.976705 5043 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-serving-cert: secret "maas-api-e2e-api-a-fab871-serving-cert" not found Nov 03 20:29:54.976838 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:54.976756 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls podName:0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:55.476739248 +0000 UTC m=+1576.943694480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls") pod "maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" (UID: "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8") : secret "maas-api-e2e-api-a-fab871-serving-cert" not found Nov 03 20:29:54.977596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:54.977573 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:55.026522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.026493 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:55.033845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.033814 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8kp\" (UniqueName: \"kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:55.055597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.054917 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7030-c72573" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" containerName="test-curl-7030-c72573" containerID="cri-o://d4f1b1431aee4475d4cc588b15f7fe9b798276b8f65e38178e8423f6d3e84d07" gracePeriod=2 Nov 03 20:29:55.164648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.164613 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:55.164813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.164675 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:55.275199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.275118 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:55.509091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.508385 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:55.527617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.527532 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\") pod \"maas-api-e2e-api-a-fab871-7885c6967d-4w8lm\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:55.654980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:55.654943 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:56.051253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:56.049716 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm"] Nov 03 20:29:56.164731 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:56.164690 5043 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:29:56.164910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:56.164744 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:29:57.040714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.040673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:57.146631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.146595 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" event={"ID":"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8","Type":"ContainerStarted","Data":"650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841"} Nov 03 20:29:57.146814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.146656 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" event={"ID":"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8","Type":"ContainerStarted","Data":"309c02d0ba513225899133c1d2df071a36d15b6d7445ae7d507681af7c1c3faf"} Nov 03 20:29:57.146814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.146684 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:29:57.148218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.148191 5043 generic.go:356] "Generic (PLEG): container finished" podID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" containerID="d4f1b1431aee4475d4cc588b15f7fe9b798276b8f65e38178e8423f6d3e84d07" exitCode=137 Nov 03 20:29:57.170326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.170300 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:57.170598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.170570 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:57.171452 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.171429 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:29:57.189881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.188606 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" podStartSLOduration=2.80202815 podStartE2EDuration="3.188587686s" totalImagesPullingTime="386.559536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:57.186963269 +0000 UTC m=+1578.653918524" watchObservedRunningTime="2026-11-03 20:29:57.188587686 +0000 UTC m=+1578.655542942" Nov 03 20:29:57.315184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.315110 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:57.411937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.411862 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f\" (UniqueName: \"kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f\") pod \"13942960-ff6b-47fd-a4d3-c2e4da89fa6d\" (UID: \"13942960-ff6b-47fd-a4d3-c2e4da89fa6d\") " Nov 03 20:29:57.428981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.428940 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f" pod "13942960-ff6b-47fd-a4d3-c2e4da89fa6d" (UID: "13942960-ff6b-47fd-a4d3-c2e4da89fa6d"). InnerVolumeSpecName "kube-api-access-b5z5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:57.545522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.545483 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5z5f\" (UniqueName: \"kubernetes.io/projected/13942960-ff6b-47fd-a4d3-c2e4da89fa6d-kube-api-access-b5z5f\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:29:57.687190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.687147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:57.973383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:57.973295 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:58.158146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:58.158108 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:58.158146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:58.158138 5043 scope.go:122] "RemoveContainer" containerID="d4f1b1431aee4475d4cc588b15f7fe9b798276b8f65e38178e8423f6d3e84d07" Nov 03 20:29:58.179671 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:58.179637 5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7030-c72573\" is forbidden: User \"system:node:ip-10-0-140-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-102.ec2.internal' and this object" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:58.680243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:58.680207 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" path="/var/lib/kubelet/pods/13942960-ff6b-47fd-a4d3-c2e4da89fa6d/volumes" Nov 03 20:29:58.688237 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:29:58.688205 5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7030-c72573\" is forbidden: User \"system:node:ip-10-0-140-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-102.ec2.internal' and this object" podUID="13942960-ff6b-47fd-a4d3-c2e4da89fa6d" pod="opendatahub/test-curl-7030-c72573" Nov 03 20:29:58.899193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:58.899157 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.435718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.435602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.569482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.569449 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.597740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.597705 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m"] Nov 03 20:29:59.604002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.603979 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.625308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.625282 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m"] Nov 03 20:29:59.631255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.631229 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7e4389\"" Nov 03 20:29:59.631255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.631234 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7e4389-dockercfg-pd6gw\"" Nov 03 20:29:59.720143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.714277 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.720143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.714393 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xvd\" (UniqueName: \"kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd\") pod \"payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.869105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.859147 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.869105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.859274 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xvd\" (UniqueName: \"kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd\") pod \"payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.869105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.860836 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.900146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.888636 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw"] Nov 03 20:29:59.900146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.895929 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:29:59.916408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.916379 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xvd\" (UniqueName: \"kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd\") pod \"payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.920003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.919939 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:29:59.922671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.922611 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw"] Nov 03 20:29:59.961351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.961037 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbg2\" (UniqueName: \"kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2\") pod \"payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:29:59.961351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:29:59.961094 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:00.062834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.062801 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbg2\" (UniqueName: \"kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2\") pod \"payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:00.063113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.062867 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:00.063916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.063880 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:00.084567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.084540 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbg2\" (UniqueName: \"kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2\") pod \"payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:00.275165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.275047 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77"] Nov 03 20:30:00.303096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.281522 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.303096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.282267 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:00.303096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.294063 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m"] Nov 03 20:30:00.312976 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.312940 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77"] Nov 03 20:30:00.370248 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.370220 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.370358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.370261 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75nd\" (UniqueName: \"kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.374643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.374618 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\"" Nov 03 20:30:00.472818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.472785 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.473349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.472836 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75nd\" (UniqueName: \"kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.473753 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.473726 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.495982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.495951 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75nd\" (UniqueName: \"kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.618370 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.618341 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:00.650165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.650136 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw"] Nov 03 20:30:00.650920 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:30:00.650893 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b72babb_158e_496a_bed3_3d6dfa3e1a6e.slice/crio-181025722d0843132d382328ef43bc4052890b4ab4a633cf6887696a934eee6b WatchSource:0}: Error finding container 181025722d0843132d382328ef43bc4052890b4ab4a633cf6887696a934eee6b: Status 404 returned error can't find the container with id 181025722d0843132d382328ef43bc4052890b4ab4a633cf6887696a934eee6b Nov 03 20:30:00.938362 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:00.938337 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77"] Nov 03 20:30:00.941196 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:30:00.941168 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d145721_a33b_45d5_8107_399a9f25f4b5.slice/crio-c92715a70bfa0e46f96c8845484abdeb10bb39e1027aa309717f89560f2e4fa9 WatchSource:0}: Error finding container c92715a70bfa0e46f96c8845484abdeb10bb39e1027aa309717f89560f2e4fa9: Status 404 returned error can't find the container with id c92715a70bfa0e46f96c8845484abdeb10bb39e1027aa309717f89560f2e4fa9 Nov 03 20:30:01.044027 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.043989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:01.190222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.190133 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" event={"ID":"2d9be17b-23c3-446b-ae0b-ca419c143d1d","Type":"ContainerStarted","Data":"2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe"} Nov 03 20:30:01.190525 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.190497 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" event={"ID":"2d9be17b-23c3-446b-ae0b-ca419c143d1d","Type":"ContainerStarted","Data":"f1f94f9b6f6dfe54c69fa66fe867d12729ee260d0867bf4d09e1d9d2d388412d"} Nov 03 20:30:01.190639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.190539 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:30:01.192980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.192951 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" event={"ID":"6b72babb-158e-496a-bed3-3d6dfa3e1a6e","Type":"ContainerStarted","Data":"fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6"} Nov 03 20:30:01.193138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.193006 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" event={"ID":"6b72babb-158e-496a-bed3-3d6dfa3e1a6e","Type":"ContainerStarted","Data":"181025722d0843132d382328ef43bc4052890b4ab4a633cf6887696a934eee6b"} Nov 03 20:30:01.193138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.193035 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:01.194791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.194768 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerStarted","Data":"c92715a70bfa0e46f96c8845484abdeb10bb39e1027aa309717f89560f2e4fa9"} Nov 03 20:30:01.230801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.230747 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" podStartSLOduration=2.230731219 podStartE2EDuration="2.230731219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.227695068 +0000 UTC m=+1582.694650322" watchObservedRunningTime="2026-11-03 20:30:01.230731219 +0000 UTC m=+1582.697686474" Nov 03 20:30:01.269960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:01.269916 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" podStartSLOduration=2.269904026 podStartE2EDuration="2.269904026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.26711009 +0000 UTC m=+1582.734065348" watchObservedRunningTime="2026-11-03 20:30:01.269904026 +0000 UTC m=+1582.736859279" Nov 03 20:30:02.715828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:02.715787 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:03.164832 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:03.164806 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:30:04.325610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:04.325564 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:05.059221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:05.059186 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:05.173490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:05.170967 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:05.245890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:05.245798 5043 generic.go:356] "Generic (PLEG): container finished" podID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerID="8e9f67d16aebee5fdd906ecbe95beb8f105d8b3cb7f271d0bd2a00ad1a0f7516" exitCode=60 Nov 03 20:30:05.246368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:05.245873 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerDied","Data":"8e9f67d16aebee5fdd906ecbe95beb8f105d8b3cb7f271d0bd2a00ad1a0f7516"} Nov 03 20:30:05.246368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:05.246350 5043 scope.go:122] "RemoveContainer" containerID="8e9f67d16aebee5fdd906ecbe95beb8f105d8b3cb7f271d0bd2a00ad1a0f7516" Nov 03 20:30:05.758369 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:30:05.753395 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:30:05.998409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:05.998327 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6"] Nov 03 20:30:06.004878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.004852 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.028994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.028968 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d1aa66-serving-cert\"" Nov 03 20:30:06.029973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.029948 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6"] Nov 03 20:30:06.083010 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.082980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:06.095556 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.095524 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhr6r\" (UniqueName: \"kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.095757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.095734 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.095953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.095932 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.096150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.096132 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.194045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.193935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:06.198133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.198063 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.198308 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.198156 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.198383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.198350 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhr6r\" (UniqueName: \"kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.198521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.198504 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.199904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.199839 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.202737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.202701 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.203618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.203598 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.220266 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.219663 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhr6r\" (UniqueName: \"kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r\") pod \"maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.257086 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.256982 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerStarted","Data":"f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74"} Nov 03 20:30:06.286457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.286420 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:06.315037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.314974 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" podStartSLOduration=3.236431661 podStartE2EDuration="6.31495584s" totalImagesPullingTime="3.078524179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:06.293239301 +0000 UTC m=+1587.760194581" watchObservedRunningTime="2026-11-03 20:30:06.31495584 +0000 UTC m=+1587.781911097" Nov 03 20:30:06.328279 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.326842 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:06.495547 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:30:06.495515 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d145721_a33b_45d5_8107_399a9f25f4b5.slice/crio-f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:30:06.674593 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:30:06.674562 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ed4fae_bc77_4a91_ab7e_5d5dfe4fbc02.slice/crio-1a48bc5761ffcbd09f11e40964229cf9213f899ceb4b288e8d5e3cd44de33a90 WatchSource:0}: Error finding container 1a48bc5761ffcbd09f11e40964229cf9213f899ceb4b288e8d5e3cd44de33a90: Status 404 returned error can't find the container with id 1a48bc5761ffcbd09f11e40964229cf9213f899ceb4b288e8d5e3cd44de33a90 Nov 03 20:30:06.683282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:06.683255 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6"] Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.272228 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" event={"ID":"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02","Type":"ContainerStarted","Data":"70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2"} Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.272298 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" event={"ID":"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02","Type":"ContainerStarted","Data":"1a48bc5761ffcbd09f11e40964229cf9213f899ceb4b288e8d5e3cd44de33a90"} Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.273472 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.275879 5043 generic.go:356] "Generic (PLEG): container finished" podID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerID="f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74" exitCode=60 Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.275965 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerDied","Data":"f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74"} Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.276006 5043 scope.go:122] "RemoveContainer" containerID="8e9f67d16aebee5fdd906ecbe95beb8f105d8b3cb7f271d0bd2a00ad1a0f7516" Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.276481 5043 scope.go:122] "RemoveContainer" containerID="f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74" Nov 03 20:30:07.281156 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:30:07.276840 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77_odh-ai-gateway-infra(2d145721-a33b-45d5-8107-399a9f25f4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" Nov 03 20:30:07.364084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.364008 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" podStartSLOduration=1.9690259110000001 podStartE2EDuration="2.363987881s" totalImagesPullingTime="394.96197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:07.323266993 +0000 UTC m=+1588.790222248" watchObservedRunningTime="2026-11-03 20:30:07.363987881 +0000 UTC m=+1588.830943136" Nov 03 20:30:07.905517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:07.905469 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:08.286958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:08.286883 5043 scope.go:122] "RemoveContainer" containerID="f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74" Nov 03 20:30:08.287919 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:30:08.287877 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77_odh-ai-gateway-infra(2d145721-a33b-45d5-8107-399a9f25f4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" Nov 03 20:30:09.525041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:09.524999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:11.179631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:11.179593 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:12.206735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:12.206704 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:30:12.207133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:12.206792 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:30:12.774625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:12.774592 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.280967 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:13.280931 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.403095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:13.402133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:14.302946 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:14.302917 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:30:14.447219 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:14.447177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:16.070039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:16.069998 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:17.562633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:17.562574 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.055107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:18.055057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.201109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:18.201057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.315162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:18.312177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.702186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:18.702149 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:20.283508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:20.283458 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:20.669804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:20.669771 5043 scope.go:122] "RemoveContainer" containerID="f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74" Nov 03 20:30:21.467539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:21.467461 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerStarted","Data":"8c583e918d427544149dc4160370c6a0ab99688f4458ea53b8fa51cedb415e21"} Nov 03 20:30:22.073610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:22.073570 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:22.478590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:22.478501 5043 generic.go:356] "Generic (PLEG): container finished" podID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerID="8c583e918d427544149dc4160370c6a0ab99688f4458ea53b8fa51cedb415e21" exitCode=60 Nov 03 20:30:22.479050 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:22.478582 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerDied","Data":"8c583e918d427544149dc4160370c6a0ab99688f4458ea53b8fa51cedb415e21"} Nov 03 20:30:22.479050 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:22.478660 5043 scope.go:122] "RemoveContainer" containerID="f01a0bda928508291be70598f5ed11e351e0a67eddc82a4137e6d3c23ef6cb74" Nov 03 20:30:22.479234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:22.479127 5043 scope.go:122] "RemoveContainer" containerID="8c583e918d427544149dc4160370c6a0ab99688f4458ea53b8fa51cedb415e21" Nov 03 20:30:22.479577 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:30:22.479551 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77_odh-ai-gateway-infra(2d145721-a33b-45d5-8107-399a9f25f4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" Nov 03 20:30:22.516555 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:22.516527 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77"] Nov 03 20:30:23.691879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.691851 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:23.734373 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.734332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:23.812533 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.812400 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca\") pod \"2d145721-a33b-45d5-8107-399a9f25f4b5\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " Nov 03 20:30:23.812813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.812785 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd\" (UniqueName: \"kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd\") pod \"2d145721-a33b-45d5-8107-399a9f25f4b5\" (UID: \"2d145721-a33b-45d5-8107-399a9f25f4b5\") " Nov 03 20:30:23.814479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.813470 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca" pod "2d145721-a33b-45d5-8107-399a9f25f4b5" (UID: "2d145721-a33b-45d5-8107-399a9f25f4b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:23.814479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.813963 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d145721-a33b-45d5-8107-399a9f25f4b5-maas-api-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:30:23.817675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.817651 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd" pod "2d145721-a33b-45d5-8107-399a9f25f4b5" (UID: "2d145721-a33b-45d5-8107-399a9f25f4b5"). InnerVolumeSpecName "kube-api-access-c75nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:23.850980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.850929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:23.923645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:23.923609 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75nd\" (UniqueName: \"kubernetes.io/projected/2d145721-a33b-45d5-8107-399a9f25f4b5-kube-api-access-c75nd\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:30:24.116725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.116688 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:24.503785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.503709 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" Nov 03 20:30:24.503785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.503733 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77" event={"ID":"2d145721-a33b-45d5-8107-399a9f25f4b5","Type":"ContainerDied","Data":"c92715a70bfa0e46f96c8845484abdeb10bb39e1027aa309717f89560f2e4fa9"} Nov 03 20:30:24.504042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.503800 5043 scope.go:122] "RemoveContainer" containerID="8c583e918d427544149dc4160370c6a0ab99688f4458ea53b8fa51cedb415e21" Nov 03 20:30:24.570228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.570192 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77"] Nov 03 20:30:24.581203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.581178 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-e3ba87-29895630-zcj77"] Nov 03 20:30:24.678925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:24.678891 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" path="/var/lib/kubelet/pods/2d145721-a33b-45d5-8107-399a9f25f4b5/volumes" Nov 03 20:30:25.516802 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:25.516760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:26.998685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:26.998639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:28.508342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:28.508295 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:30.083497 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.083443 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp"] Nov 03 20:30:30.085443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.085411 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:30:30.085443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.085439 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:30:30.085648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.085492 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:30:30.085648 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.085500 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:30:30.085760 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.085743 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:30:30.085816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.085777 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:30:30.091795 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.091770 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.116555 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.116470 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp"] Nov 03 20:30:30.138710 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.118761 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7e4389-serving-cert\"" Nov 03 20:30:30.165487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.165426 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:30.241960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.241921 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.242178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.242115 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.242178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.242146 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4w4k\" (UniqueName: \"kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.242340 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.242178 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.287218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.286349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:30.347138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.346022 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.347138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.346280 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.347138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.346315 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4w4k\" (UniqueName: \"kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.347138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.346349 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.347138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.346806 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.359913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.359880 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.361892 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.361863 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.366435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.366411 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4w4k\" (UniqueName: \"kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k\") pod \"maas-api-e2e-api-b-7e4389-c69667cfb-djbdp\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.394250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.394181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:30.425316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.425285 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:30.743122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:30.743057 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp"] Nov 03 20:30:30.745568 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:30:30.745540 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277c75ae_0568_4c6c_bcfb_b574312c43ca.slice/crio-6c1713e1151f101402967817061f0f37a10723a4ffdab91f4e76bd0ecfa4d3b0 WatchSource:0}: Error finding container 6c1713e1151f101402967817061f0f37a10723a4ffdab91f4e76bd0ecfa4d3b0: Status 404 returned error can't find the container with id 6c1713e1151f101402967817061f0f37a10723a4ffdab91f4e76bd0ecfa4d3b0 Nov 03 20:30:31.065664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:31.065590 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:31.573953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:31.573921 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" event={"ID":"277c75ae-0568-4c6c-bcfb-b574312c43ca","Type":"ContainerStarted","Data":"63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa"} Nov 03 20:30:31.574359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:31.573972 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" event={"ID":"277c75ae-0568-4c6c-bcfb-b574312c43ca","Type":"ContainerStarted","Data":"6c1713e1151f101402967817061f0f37a10723a4ffdab91f4e76bd0ecfa4d3b0"} Nov 03 20:30:31.574359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:31.574094 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:31.611123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:31.611051 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:31.618204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:31.618148 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" podStartSLOduration=1.296178431 podStartE2EDuration="1.618129592s" totalImagesPullingTime="321.951161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:31.613286731 +0000 UTC m=+1613.080242003" watchObservedRunningTime="2026-11-03 20:30:31.618129592 +0000 UTC m=+1613.085084847" Nov 03 20:30:32.127205 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:32.127163 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:33.827856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:33.827812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:34.393857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:34.393825 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.077084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:35.077045 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.464448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:35.464357 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.595335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:35.595281 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.725430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:35.725303 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.739886 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:30:35.739847 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:30:37.228036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:37.228000 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:37.612307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:37.612271 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:30:39.010710 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:39.010675 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.820371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:39.820336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.921959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:39.921919 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:40.911489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:40.911450 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:41.545975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:41.545926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:41.643608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:41.643569 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:42.786608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:42.786571 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:44.569920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:44.569878 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:44.680900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:44.680788 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:46.306957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:46.306913 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:47.932815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:47.932775 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:49.873337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:49.872868 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:51.773740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:51.773689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:52.598097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:52.597161 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:30:52.598097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:52.597271 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:53.079639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:53.079604 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:30:53.080354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:53.080294 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:53.446470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:53.446382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:53.923144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:53.923066 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:54.122685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:54.122644 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:54.290690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:54.290607 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:54.573549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:54.573456 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:56.084101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:56.084044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.479603 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:57.479551 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.612416 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:57.612375 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:58.049079 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:58.049043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:59.710645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:30:59.710604 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:00.837003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:00.836964 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:01.516575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:01.516538 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:02.954550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:02.954512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.126922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:03.126881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.331229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:03.331186 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.488180 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:03.488146 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:04.051029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:04.050989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:04.315564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:04.315474 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:04.549276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:04.549239 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:04.667690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:04.667654 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:05.370919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:05.370886 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:05.635104 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:31:05.635008 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:31:07.161719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:07.161677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:08.374139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:08.374097 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:08.497541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:08.497497 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:09.070863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:09.070822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:10.941884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:10.941830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:11.042455 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:11.042415 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:11.152216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:11.152184 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:11.238624 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:11.238541 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx"] Nov 03 20:31:11.239165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:11.238829 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" containerID="cri-o://2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e" gracePeriod=30 Nov 03 20:31:11.299035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:11.298996 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:12.806250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:12.806212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:12.892392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:12.892353 5043 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:31:12.892742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:12.892708 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:31:14.109935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:14.109890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:14.490532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:14.490412 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:14.589903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:14.589865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:14.697106 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:14.697052 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:16.627199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.627169 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:31:16.716233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.716184 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:16.718395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718364 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718456 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718518 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718618 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718677 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:16.718715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718682 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718776 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:16.718825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718800 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718861 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.718928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718898 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:16.719030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.718940 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.719101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719045 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data\") pod \"e3ae8950-3221-420c-b2eb-fd4035002a6f\" (UID: \"e3ae8950-3221-420c-b2eb-fd4035002a6f\") " Nov 03 20:31:16.719343 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719310 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:16.719800 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719772 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:16.719934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719857 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.719934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719882 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.719934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719899 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ae8950-3221-420c-b2eb-fd4035002a6f-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.719934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.719915 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.721524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.721495 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:16.721619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.721538 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:16.721986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.721960 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "kube-api-access-nxk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:16.722683 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.722658 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy" pod "e3ae8950-3221-420c-b2eb-fd4035002a6f" (UID: "e3ae8950-3221-420c-b2eb-fd4035002a6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:16.828090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.826517 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxk6d\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-kube-api-access-nxk6d\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.828090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.826553 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.828090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.826567 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.828090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.826581 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:16.828090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:16.826594 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ae8950-3221-420c-b2eb-fd4035002a6f-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:31:17.272273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.272166 5043 generic.go:356] "Generic (PLEG): container finished" podID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerID="2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e" exitCode=0 Nov 03 20:31:17.272273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.272253 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" Nov 03 20:31:17.272530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.272349 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" event={"ID":"e3ae8950-3221-420c-b2eb-fd4035002a6f","Type":"ContainerDied","Data":"2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e"} Nov 03 20:31:17.272530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.272431 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx" event={"ID":"e3ae8950-3221-420c-b2eb-fd4035002a6f","Type":"ContainerDied","Data":"a40101d3ecee96ea29dfd232dda6c47fd3b4ce2d8cbd02467e96e25efeccc4e2"} Nov 03 20:31:17.272530 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.272457 5043 scope.go:122] "RemoveContainer" containerID="2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e" Nov 03 20:31:17.286155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.286134 5043 scope.go:122] "RemoveContainer" containerID="2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e" Nov 03 20:31:17.286468 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:31:17.286447 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e\": container with ID starting with 2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e not found: ID does not exist" containerID="2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e" Nov 03 20:31:17.286528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.286477 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e"} err="failed to get container status \"2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e\": rpc error: code = NotFound desc = could not find container \"2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e\": container with ID starting with 2d8ea22ae517602da4fc0a7c54c54bc1fb85a49c43135727011be16005eb6a6e not found: ID does not exist" Nov 03 20:31:17.332609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.332580 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx"] Nov 03 20:31:17.336739 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:17.336716 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zffrx"] Nov 03 20:31:18.401340 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:18.401304 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:18.679745 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:18.679663 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" path="/var/lib/kubelet/pods/e3ae8950-3221-420c-b2eb-fd4035002a6f/volumes" Nov 03 20:31:19.959421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:19.959388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:21.537280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:21.537200 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:23.116297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:23.116257 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:24.863635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:24.863604 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:25.712607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:25.712573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:25.816996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:25.816959 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:26.643959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:26.643917 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:28.166734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:28.166693 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:29.715036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:29.714999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:31.392941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:31.392897 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:31.725563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:31.725465 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:32.917650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:32.917609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:34.701302 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:34.701253 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:35.853099 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:31:35.849689 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:31:36.851843 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:36.851810 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:38.459940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:38.459902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:40.027120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:40.027067 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:40.128012 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:40.127974 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:41.697136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:41.697101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:43.194019 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:43.193982 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:44.856928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:44.856894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:46.302489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:46.302301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:46.818032 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:46.817992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:46.940488 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:46.940452 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:47.052041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:47.052003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:47.406007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:47.405967 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:48.817144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:48.817107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:50.454855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:50.454812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:51.122028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:51.121989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.179107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:52.179029 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.276531 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:54.276469 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.408569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:54.408533 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.601594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:54.601552 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.562704 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:56.562667 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.116709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:58.116673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:59.676119 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:31:59.676057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.170933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:01.170900 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.785312 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:02.785276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:03.154586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:03.154537 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:03.263283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:03.256147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:04.427295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:04.427259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:04.639320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:04.639285 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs"] Nov 03 20:32:04.747274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:04.747194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:04.957345 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:04.957301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.609359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:05.609321 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.639407 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:05.639379 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:32:05.727290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:05.726702 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:06.144164 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:06.144128 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:06.285998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:06.285363 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm"] Nov 03 20:32:06.307280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:06.307246 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65"] Nov 03 20:32:06.437135 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:06.437029 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:06.536647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:06.536610 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.729259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:07.729225 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.401544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:09.401500 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.639809 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:09.639740 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" podUID="098e1578-d007-4299-82cf-d9e05369f350" containerName="payload-pre-processing" containerID="cri-o://0b0686c763816349132532e31a7005c8d6c2459b04a21634eb061988dc37720c" gracePeriod=25 Nov 03 20:32:09.873954 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:09.873915 5043 generic.go:356] "Generic (PLEG): container finished" podID="098e1578-d007-4299-82cf-d9e05369f350" containerID="0b0686c763816349132532e31a7005c8d6c2459b04a21634eb061988dc37720c" exitCode=0 Nov 03 20:32:09.874150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:09.873984 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" event={"ID":"098e1578-d007-4299-82cf-d9e05369f350","Type":"ContainerDied","Data":"0b0686c763816349132532e31a7005c8d6c2459b04a21634eb061988dc37720c"} Nov 03 20:32:09.943392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:09.943362 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:32:10.034165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.034130 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume\") pod \"098e1578-d007-4299-82cf-d9e05369f350\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " Nov 03 20:32:10.034365 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.034285 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4\" (UniqueName: \"kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4\") pod \"098e1578-d007-4299-82cf-d9e05369f350\" (UID: \"098e1578-d007-4299-82cf-d9e05369f350\") " Nov 03 20:32:10.034610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.034582 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume" pod "098e1578-d007-4299-82cf-d9e05369f350" (UID: "098e1578-d007-4299-82cf-d9e05369f350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:10.035212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.035187 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098e1578-d007-4299-82cf-d9e05369f350-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:10.037607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.037571 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4" pod "098e1578-d007-4299-82cf-d9e05369f350" (UID: "098e1578-d007-4299-82cf-d9e05369f350"). InnerVolumeSpecName "kube-api-access-j87n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:10.136607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.136579 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87n4\" (UniqueName: \"kubernetes.io/projected/098e1578-d007-4299-82cf-d9e05369f350-kube-api-access-j87n4\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:10.811132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.811052 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.884777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.884741 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" Nov 03 20:32:10.884777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.884759 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs" event={"ID":"098e1578-d007-4299-82cf-d9e05369f350","Type":"ContainerDied","Data":"187172c0c7249cf963fae30fca344a8c0520114310c2a8e442667b7b28afaba2"} Nov 03 20:32:10.885024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.884824 5043 scope.go:122] "RemoveContainer" containerID="0b0686c763816349132532e31a7005c8d6c2459b04a21634eb061988dc37720c" Nov 03 20:32:10.935202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.935153 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs"] Nov 03 20:32:10.946241 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:10.946182 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-de82c8-58bbb7c49f-jfrcs"] Nov 03 20:32:11.293166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.287890 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" podUID="1a7473c1-6c0e-4644-89dd-3124415127b2" containerName="payload-processing" containerID="cri-o://2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e" gracePeriod=25 Nov 03 20:32:11.311594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.310550 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" podUID="ddad4d1d-a61a-457e-8523-b4021250089f" containerName="payload-pre-processing" containerID="cri-o://460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2" gracePeriod=25 Nov 03 20:32:11.330614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.330534 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:11.431186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.431091 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:11.553803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.550138 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:11.781969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.781942 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:32:11.791679 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.791638 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume\") pod \"1a7473c1-6c0e-4644-89dd-3124415127b2\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " Nov 03 20:32:11.792255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.792227 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw\" (UniqueName: \"kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw\") pod \"1a7473c1-6c0e-4644-89dd-3124415127b2\" (UID: \"1a7473c1-6c0e-4644-89dd-3124415127b2\") " Nov 03 20:32:11.793243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.792048 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume" pod "1a7473c1-6c0e-4644-89dd-3124415127b2" (UID: "1a7473c1-6c0e-4644-89dd-3124415127b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:11.807057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.806999 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw" pod "1a7473c1-6c0e-4644-89dd-3124415127b2" (UID: "1a7473c1-6c0e-4644-89dd-3124415127b2"). InnerVolumeSpecName "kube-api-access-4d4tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:11.895185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.895146 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473c1-6c0e-4644-89dd-3124415127b2-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:11.895185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.895185 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4tw\" (UniqueName: \"kubernetes.io/projected/1a7473c1-6c0e-4644-89dd-3124415127b2-kube-api-access-4d4tw\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:11.895660 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.895187 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:32:11.900948 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.900919 5043 generic.go:356] "Generic (PLEG): container finished" podID="ddad4d1d-a61a-457e-8523-b4021250089f" containerID="460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2" exitCode=0 Nov 03 20:32:11.901223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.900968 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" Nov 03 20:32:11.901809 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.900983 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" event={"ID":"ddad4d1d-a61a-457e-8523-b4021250089f","Type":"ContainerDied","Data":"460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2"} Nov 03 20:32:11.901983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.901960 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65" event={"ID":"ddad4d1d-a61a-457e-8523-b4021250089f","Type":"ContainerDied","Data":"6bee8580046cdec1731ef9d63ebb877b1864d7fa582de9a5ed242dd34adc3441"} Nov 03 20:32:11.902098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.902003 5043 scope.go:122] "RemoveContainer" containerID="460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2" Nov 03 20:32:11.904334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.904035 5043 generic.go:356] "Generic (PLEG): container finished" podID="1a7473c1-6c0e-4644-89dd-3124415127b2" containerID="2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e" exitCode=0 Nov 03 20:32:11.904334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.904095 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" event={"ID":"1a7473c1-6c0e-4644-89dd-3124415127b2","Type":"ContainerDied","Data":"2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e"} Nov 03 20:32:11.904334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.904169 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" event={"ID":"1a7473c1-6c0e-4644-89dd-3124415127b2","Type":"ContainerDied","Data":"b8b551dcca890649a4c733d3469d590e697a4b43a568b2c879f508b442ae8390"} Nov 03 20:32:11.904526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.904366 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm" Nov 03 20:32:11.920487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.919960 5043 scope.go:122] "RemoveContainer" containerID="460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2" Nov 03 20:32:11.920619 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:11.920489 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2\": container with ID starting with 460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2 not found: ID does not exist" containerID="460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2" Nov 03 20:32:11.920619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.920524 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2"} err="failed to get container status \"460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2\": rpc error: code = NotFound desc = could not find container \"460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2\": container with ID starting with 460bef8c87d4acaf53564e99f370a82b4a034b20727195bb62c506969b7aedc2 not found: ID does not exist" Nov 03 20:32:11.920619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.920548 5043 scope.go:122] "RemoveContainer" containerID="2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e" Nov 03 20:32:11.937923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.937907 5043 scope.go:122] "RemoveContainer" containerID="2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e" Nov 03 20:32:11.941013 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:11.939645 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e\": container with ID starting with 2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e not found: ID does not exist" containerID="2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e" Nov 03 20:32:11.941013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.939682 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e"} err="failed to get container status \"2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e\": rpc error: code = NotFound desc = could not find container \"2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e\": container with ID starting with 2a2746d1a156ed70b5eff558fd0c7530398ad4d64294255342f925a42fd3ac1e not found: ID does not exist" Nov 03 20:32:11.966229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.966201 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:11.974169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.972861 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm"] Nov 03 20:32:11.979159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.979132 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b917c2-f4f8577cb-xvbgm"] Nov 03 20:32:11.996544 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.996512 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2\" (UniqueName: \"kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2\") pod \"ddad4d1d-a61a-457e-8523-b4021250089f\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " Nov 03 20:32:11.996825 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.996801 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume\") pod \"ddad4d1d-a61a-457e-8523-b4021250089f\" (UID: \"ddad4d1d-a61a-457e-8523-b4021250089f\") " Nov 03 20:32:11.998143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:11.998117 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume" pod "ddad4d1d-a61a-457e-8523-b4021250089f" (UID: "ddad4d1d-a61a-457e-8523-b4021250089f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:12.000529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.000487 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2" pod "ddad4d1d-a61a-457e-8523-b4021250089f" (UID: "ddad4d1d-a61a-457e-8523-b4021250089f"). InnerVolumeSpecName "kube-api-access-vv8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:12.099022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.098982 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8g2\" (UniqueName: \"kubernetes.io/projected/ddad4d1d-a61a-457e-8523-b4021250089f-kube-api-access-vv8g2\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:12.099022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.099024 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad4d1d-a61a-457e-8523-b4021250089f-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:12.249458 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.249425 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65"] Nov 03 20:32:12.253766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.253738 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b917c2-64f7bbfbdc-hff65"] Nov 03 20:32:12.679667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.679583 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098e1578-d007-4299-82cf-d9e05369f350" path="/var/lib/kubelet/pods/098e1578-d007-4299-82cf-d9e05369f350/volumes" Nov 03 20:32:12.680234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.680215 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7473c1-6c0e-4644-89dd-3124415127b2" path="/var/lib/kubelet/pods/1a7473c1-6c0e-4644-89dd-3124415127b2/volumes" Nov 03 20:32:12.680858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:12.680837 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddad4d1d-a61a-457e-8523-b4021250089f" path="/var/lib/kubelet/pods/ddad4d1d-a61a-457e-8523-b4021250089f/volumes" Nov 03 20:32:13.410625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:13.410589 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:15.281465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:15.281426 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:16.061404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:16.061368 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz"] Nov 03 20:32:16.061613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:16.061575 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" containerID="cri-o://f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01" gracePeriod=30 Nov 03 20:32:17.022999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:17.022952 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.114218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.114179 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.215771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.215735 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.400488 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.400397 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw"] Nov 03 20:32:18.400755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.400721 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" containerID="cri-o://a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7" gracePeriod=30 Nov 03 20:32:18.452087 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.448545 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.548468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.548427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.950653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.950613 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:18.950858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:18.950679 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:19.683759 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:19.683724 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.092914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:20.092881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.335774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:20.335735 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.643509 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:20.643448 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.803919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:20.803886 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.368053 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.368029 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:32:21.414996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.414962 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415015 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415059 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415134 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415256 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415298 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415404 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.415486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415431 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415516 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415606 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv\") pod \"25b66086-db00-4280-8a94-e515abf810ab\" (UID: \"25b66086-db00-4280-8a94-e515abf810ab\") " Nov 03 20:32:21.415871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415844 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.415947 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.415874 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:21.416182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.416161 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.416264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.416193 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b66086-db00-4280-8a94-e515abf810ab-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.416264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.416214 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.416264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.416230 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.416504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.416486 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.418144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.418118 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.418797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.418770 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:21.418797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.418781 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:21.418930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.418901 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv" pod "25b66086-db00-4280-8a94-e515abf810ab" (UID: "25b66086-db00-4280-8a94-e515abf810ab"). InnerVolumeSpecName "kube-api-access-wfpzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:21.518006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.517975 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b66086-db00-4280-8a94-e515abf810ab-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.518006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.518001 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.518006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.518010 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.518283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.518018 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.518283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.518048 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b66086-db00-4280-8a94-e515abf810ab-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:21.518283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:21.518058 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpzv\" (UniqueName: \"kubernetes.io/projected/25b66086-db00-4280-8a94-e515abf810ab-kube-api-access-wfpzv\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:22.047883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.047839 5043 generic.go:356] "Generic (PLEG): container finished" podID="25b66086-db00-4280-8a94-e515abf810ab" containerID="f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01" exitCode=0 Nov 03 20:32:22.048385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.047918 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" event={"ID":"25b66086-db00-4280-8a94-e515abf810ab","Type":"ContainerDied","Data":"f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01"} Nov 03 20:32:22.048385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.047917 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" Nov 03 20:32:22.048385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.047988 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz" event={"ID":"25b66086-db00-4280-8a94-e515abf810ab","Type":"ContainerDied","Data":"9f86ce2fe4a8e4df7dc4759a8d7846627b96c642b2bb23b94f5227b929f6a61c"} Nov 03 20:32:22.048385 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.048020 5043 scope.go:122] "RemoveContainer" containerID="f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01" Nov 03 20:32:22.067115 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.067089 5043 scope.go:122] "RemoveContainer" containerID="f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01" Nov 03 20:32:22.067424 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:22.067401 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01\": container with ID starting with f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01 not found: ID does not exist" containerID="f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01" Nov 03 20:32:22.067516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.067431 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01"} err="failed to get container status \"f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01\": rpc error: code = NotFound desc = could not find container \"f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01\": container with ID starting with f7b7e0f70cbb796b217a0e6d22b311e8569ba9b815eeaecaa447a64995f3de01 not found: ID does not exist" Nov 03 20:32:22.106779 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.106728 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz"] Nov 03 20:32:22.111678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.111651 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.116156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.116130 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-97jrz"] Nov 03 20:32:22.679559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:22.679519 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b66086-db00-4280-8a94-e515abf810ab" path="/var/lib/kubelet/pods/25b66086-db00-4280-8a94-e515abf810ab/volumes" Nov 03 20:32:23.718940 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.718915 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:32:23.760023 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.759992 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.760204 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760061 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.760329 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760241 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.760329 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760277 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760357 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760402 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760485 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760490 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760524 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760604 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs\") pod \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\" (UID: \"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97\") " Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760672 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.761188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.760923 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:23.761599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.761512 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.761599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.761571 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.761599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.761597 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.761746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.761610 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.761746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.761690 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.762983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.762953 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.763127 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.763105 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:23.763782 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.763758 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "kube-api-access-5d9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:23.763921 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.763902 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo" pod "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" (UID: "7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:23.792304 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.787424 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:23.868572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.864056 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.868572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.864105 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.868572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.864121 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9nf\" (UniqueName: \"kubernetes.io/projected/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-kube-api-access-5d9nf\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.868572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.864136 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.868572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.864150 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:23.868572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:23.864162 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:24.072139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.071571 5043 generic.go:356] "Generic (PLEG): container finished" podID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerID="a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7" exitCode=0 Nov 03 20:32:24.072139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.071642 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" Nov 03 20:32:24.072139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.071857 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" event={"ID":"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97","Type":"ContainerDied","Data":"a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7"} Nov 03 20:32:24.072139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.071912 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw" event={"ID":"7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97","Type":"ContainerDied","Data":"7e6cd096c74390b1d3027eeda279e5b6293cb7e28bfa8d1d51c106488dc1f49d"} Nov 03 20:32:24.072139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.071948 5043 scope.go:122] "RemoveContainer" containerID="a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7" Nov 03 20:32:24.089549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.089521 5043 scope.go:122] "RemoveContainer" containerID="a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7" Nov 03 20:32:24.089913 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:24.089886 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7\": container with ID starting with a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7 not found: ID does not exist" containerID="a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7" Nov 03 20:32:24.090012 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.089919 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7"} err="failed to get container status \"a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7\": rpc error: code = NotFound desc = could not find container \"a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7\": container with ID starting with a85e97b46ea4d551d2fb901fcdf47c6bcba077b5bff32aae07b4b3c66691e1e7 not found: ID does not exist" Nov 03 20:32:24.126549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.126516 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw"] Nov 03 20:32:24.130982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.130958 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-pdpsw"] Nov 03 20:32:24.499120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.499067 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.607646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.607609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.682417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:24.682386 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" path="/var/lib/kubelet/pods/7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97/volumes" Nov 03 20:32:25.831953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:25.831915 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:26.356150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:26.356103 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:26.545913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:26.545880 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:27.751866 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:27.751832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.552713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:28.552680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.652880 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:28.652846 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:29.398638 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:29.398598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:29.426896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:29.426864 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw"] Nov 03 20:32:29.461553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:29.461524 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m"] Nov 03 20:32:29.561590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:29.561553 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:29.665211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:29.665126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:30.892816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:30.892774 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:31.825534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:31.825499 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.128746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.128655 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8"] Nov 03 20:32:32.171596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.171556 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" podUID="2045fa93-9dec-4d08-9d76-2dc4122e1306" containerName="maas-api" containerID="cri-o://9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c" gracePeriod=30 Nov 03 20:32:32.196427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.196392 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.652774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.652744 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:32:32.694670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.694550 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.747095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.745323 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls\") pod \"2045fa93-9dec-4d08-9d76-2dc4122e1306\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " Nov 03 20:32:32.747095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.745458 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs\" (UniqueName: \"kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs\") pod \"2045fa93-9dec-4d08-9d76-2dc4122e1306\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " Nov 03 20:32:32.747095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.745566 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs\") pod \"2045fa93-9dec-4d08-9d76-2dc4122e1306\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " Nov 03 20:32:32.747095 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.745632 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca\") pod \"2045fa93-9dec-4d08-9d76-2dc4122e1306\" (UID: \"2045fa93-9dec-4d08-9d76-2dc4122e1306\") " Nov 03 20:32:32.754881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.749486 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca" pod "2045fa93-9dec-4d08-9d76-2dc4122e1306" (UID: "2045fa93-9dec-4d08-9d76-2dc4122e1306"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:32.761277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.761242 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs" pod "2045fa93-9dec-4d08-9d76-2dc4122e1306" (UID: "2045fa93-9dec-4d08-9d76-2dc4122e1306"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:32.776132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.775373 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls" pod "2045fa93-9dec-4d08-9d76-2dc4122e1306" (UID: "2045fa93-9dec-4d08-9d76-2dc4122e1306"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:32.778307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.778274 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs" pod "2045fa93-9dec-4d08-9d76-2dc4122e1306" (UID: "2045fa93-9dec-4d08-9d76-2dc4122e1306"). InnerVolumeSpecName "kube-api-access-2wcgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:32.799040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.795803 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.863037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.862952 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wcgs\" (UniqueName: \"kubernetes.io/projected/2045fa93-9dec-4d08-9d76-2dc4122e1306-kube-api-access-2wcgs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:32.863037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.862987 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:32.863037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.863002 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2045fa93-9dec-4d08-9d76-2dc4122e1306-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:32.863037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.863013 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2045fa93-9dec-4d08-9d76-2dc4122e1306-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:32.912137 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:32.910714 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:33.180450 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.180365 5043 generic.go:356] "Generic (PLEG): container finished" podID="2045fa93-9dec-4d08-9d76-2dc4122e1306" containerID="9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c" exitCode=0 Nov 03 20:32:33.180450 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.180427 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" event={"ID":"2045fa93-9dec-4d08-9d76-2dc4122e1306","Type":"ContainerDied","Data":"9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c"} Nov 03 20:32:33.180909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.180439 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" Nov 03 20:32:33.180909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.180539 5043 scope.go:122] "RemoveContainer" containerID="9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c" Nov 03 20:32:33.180909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.180523 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8" event={"ID":"2045fa93-9dec-4d08-9d76-2dc4122e1306","Type":"ContainerDied","Data":"d82945ad2af3770f5c666e3930dc06a76ffd84b52e814995d7eea9d6e6bcc740"} Nov 03 20:32:33.192599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.192576 5043 scope.go:122] "RemoveContainer" containerID="9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c" Nov 03 20:32:33.192878 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:33.192859 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c\": container with ID starting with 9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c not found: ID does not exist" containerID="9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c" Nov 03 20:32:33.192933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.192887 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c"} err="failed to get container status \"9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c\": rpc error: code = NotFound desc = could not find container \"9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c\": container with ID starting with 9dfae4078a167219d9d26d46ed77259e5a7f2c926bd292b4952d624413e40b4c not found: ID does not exist" Nov 03 20:32:33.268188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.268165 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8"] Nov 03 20:32:33.278280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.278256 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-de82c8-5678f787cf-dh8b8"] Nov 03 20:32:33.361729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:33.361695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:34.324966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.324912 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx"] Nov 03 20:32:34.325575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.325543 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" podUID="8f4eced3-9286-4b3a-9d75-0fe80735f8b5" containerName="maas-api" containerID="cri-o://d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5" gracePeriod=30 Nov 03 20:32:34.378227 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.378197 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:34.428099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.428047 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" podUID="6b72babb-158e-496a-bed3-3d6dfa3e1a6e" containerName="payload-processing" containerID="cri-o://fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6" gracePeriod=25 Nov 03 20:32:34.462569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.462509 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" podUID="2d9be17b-23c3-446b-ae0b-ca419c143d1d" containerName="payload-pre-processing" containerID="cri-o://2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe" gracePeriod=25 Nov 03 20:32:34.854229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.854152 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:32:34.858193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.858165 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2045fa93-9dec-4d08-9d76-2dc4122e1306" path="/var/lib/kubelet/pods/2045fa93-9dec-4d08-9d76-2dc4122e1306/volumes" Nov 03 20:32:34.976537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.976490 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs\") pod \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " Nov 03 20:32:34.976729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:34.976559 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6\" (UniqueName: \"kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6\") pod \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " Nov 03 20:32:35.002213 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.002147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.005484 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.005444 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs" pod "8f4eced3-9286-4b3a-9d75-0fe80735f8b5" (UID: "8f4eced3-9286-4b3a-9d75-0fe80735f8b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:35.015012 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.014956 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6" pod "8f4eced3-9286-4b3a-9d75-0fe80735f8b5" (UID: "8f4eced3-9286-4b3a-9d75-0fe80735f8b5"). InnerVolumeSpecName "kube-api-access-8mkk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.071642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.071617 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:32:35.079159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.079061 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca\") pod \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " Nov 03 20:32:35.079331 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.079312 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume\") pod \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " Nov 03 20:32:35.079479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.079455 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca" pod "8f4eced3-9286-4b3a-9d75-0fe80735f8b5" (UID: "8f4eced3-9286-4b3a-9d75-0fe80735f8b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:35.079732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.079710 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume" pod "6b72babb-158e-496a-bed3-3d6dfa3e1a6e" (UID: "6b72babb-158e-496a-bed3-3d6dfa3e1a6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:35.079789 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.079760 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2\" (UniqueName: \"kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2\") pod \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\" (UID: \"6b72babb-158e-496a-bed3-3d6dfa3e1a6e\") " Nov 03 20:32:35.079851 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.079807 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls\") pod \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\" (UID: \"8f4eced3-9286-4b3a-9d75-0fe80735f8b5\") " Nov 03 20:32:35.081535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.081505 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.081664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.081537 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkk6\" (UniqueName: \"kubernetes.io/projected/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-kube-api-access-8mkk6\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.081664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.081553 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.081664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.081569 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.082400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.082376 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls" pod "8f4eced3-9286-4b3a-9d75-0fe80735f8b5" (UID: "8f4eced3-9286-4b3a-9d75-0fe80735f8b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:35.084423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.084398 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2" pod "6b72babb-158e-496a-bed3-3d6dfa3e1a6e" (UID: "6b72babb-158e-496a-bed3-3d6dfa3e1a6e"). InnerVolumeSpecName "kube-api-access-slbg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.099910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.099862 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.177086 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.177035 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:32:35.183111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.183032 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbg2\" (UniqueName: \"kubernetes.io/projected/6b72babb-158e-496a-bed3-3d6dfa3e1a6e-kube-api-access-slbg2\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.183111 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.183083 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4eced3-9286-4b3a-9d75-0fe80735f8b5-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.204545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.204371 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.210553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.210526 5043 generic.go:356] "Generic (PLEG): container finished" podID="8f4eced3-9286-4b3a-9d75-0fe80735f8b5" containerID="d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5" exitCode=0 Nov 03 20:32:35.210695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.210580 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" Nov 03 20:32:35.210695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.210609 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" event={"ID":"8f4eced3-9286-4b3a-9d75-0fe80735f8b5","Type":"ContainerDied","Data":"d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5"} Nov 03 20:32:35.210695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.210665 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx" event={"ID":"8f4eced3-9286-4b3a-9d75-0fe80735f8b5","Type":"ContainerDied","Data":"852c14077a391de5f27bb66cc5431050886734b80421cb1b5fb9daf5c7be2595"} Nov 03 20:32:35.210695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.210687 5043 scope.go:122] "RemoveContainer" containerID="d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5" Nov 03 20:32:35.212508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.212485 5043 generic.go:356] "Generic (PLEG): container finished" podID="2d9be17b-23c3-446b-ae0b-ca419c143d1d" containerID="2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe" exitCode=0 Nov 03 20:32:35.212595 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.212563 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" event={"ID":"2d9be17b-23c3-446b-ae0b-ca419c143d1d","Type":"ContainerDied","Data":"2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe"} Nov 03 20:32:35.212678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.212600 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" event={"ID":"2d9be17b-23c3-446b-ae0b-ca419c143d1d","Type":"ContainerDied","Data":"f1f94f9b6f6dfe54c69fa66fe867d12729ee260d0867bf4d09e1d9d2d388412d"} Nov 03 20:32:35.212678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.212655 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m" Nov 03 20:32:35.214480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.214459 5043 generic.go:356] "Generic (PLEG): container finished" podID="6b72babb-158e-496a-bed3-3d6dfa3e1a6e" containerID="fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6" exitCode=0 Nov 03 20:32:35.214604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.214516 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" Nov 03 20:32:35.214604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.214583 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" event={"ID":"6b72babb-158e-496a-bed3-3d6dfa3e1a6e","Type":"ContainerDied","Data":"fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6"} Nov 03 20:32:35.214690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.214622 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw" event={"ID":"6b72babb-158e-496a-bed3-3d6dfa3e1a6e","Type":"ContainerDied","Data":"181025722d0843132d382328ef43bc4052890b4ab4a633cf6887696a934eee6b"} Nov 03 20:32:35.232116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.230285 5043 scope.go:122] "RemoveContainer" containerID="d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5" Nov 03 20:32:35.234004 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:35.232608 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5\": container with ID starting with d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5 not found: ID does not exist" containerID="d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5" Nov 03 20:32:35.234004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.232645 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5"} err="failed to get container status \"d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5\": rpc error: code = NotFound desc = could not find container \"d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5\": container with ID starting with d6346110c652d9c4258a2fc4b6a120b66961ba1d9bfff965bf201b2cf504b9b5 not found: ID does not exist" Nov 03 20:32:35.234004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.232669 5043 scope.go:122] "RemoveContainer" containerID="2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe" Nov 03 20:32:35.254543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.254512 5043 scope.go:122] "RemoveContainer" containerID="2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe" Nov 03 20:32:35.255426 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:35.254980 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe\": container with ID starting with 2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe not found: ID does not exist" containerID="2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe" Nov 03 20:32:35.255426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.255025 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe"} err="failed to get container status \"2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe\": rpc error: code = NotFound desc = could not find container \"2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe\": container with ID starting with 2dc020397d9cf34dafebe4a285ae238e466e5a5b06eecabcea542c5a0f827ebe not found: ID does not exist" Nov 03 20:32:35.255426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.255053 5043 scope.go:122] "RemoveContainer" containerID="fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6" Nov 03 20:32:35.298451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.298387 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd\" (UniqueName: \"kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd\") pod \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " Nov 03 20:32:35.298662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.298547 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume\") pod \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\" (UID: \"2d9be17b-23c3-446b-ae0b-ca419c143d1d\") " Nov 03 20:32:35.299450 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.299402 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume" pod "2d9be17b-23c3-446b-ae0b-ca419c143d1d" (UID: "2d9be17b-23c3-446b-ae0b-ca419c143d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:35.301520 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.301494 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd" pod "2d9be17b-23c3-446b-ae0b-ca419c143d1d" (UID: "2d9be17b-23c3-446b-ae0b-ca419c143d1d"). InnerVolumeSpecName "kube-api-access-v5xvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.306335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.306315 5043 scope.go:122] "RemoveContainer" containerID="fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6" Nov 03 20:32:35.307002 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:35.306972 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6\": container with ID starting with fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6 not found: ID does not exist" containerID="fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6" Nov 03 20:32:35.307175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.307150 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6"} err="failed to get container status \"fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6\": rpc error: code = NotFound desc = could not find container \"fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6\": container with ID starting with fb0d1166e61213026604a9dd2eed90634283415ef08637bdd6fd41712cfbd6d6 not found: ID does not exist" Nov 03 20:32:35.328392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.328361 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx"] Nov 03 20:32:35.334805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.334785 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b917c2-74d9bfbb6b-xzgrx"] Nov 03 20:32:35.366139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.363019 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw"] Nov 03 20:32:35.368726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.368704 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7e4389-ddd764bdd-q9djw"] Nov 03 20:32:35.400944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.400911 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9be17b-23c3-446b-ae0b-ca419c143d1d-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.400944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.400943 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xvd\" (UniqueName: \"kubernetes.io/projected/2d9be17b-23c3-446b-ae0b-ca419c143d1d-kube-api-access-v5xvd\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:35.559279 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.559248 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.561537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.561512 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m"] Nov 03 20:32:35.565280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:35.565260 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7e4389-bc6d454d7-r7z6m"] Nov 03 20:32:35.633083 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:35.633037 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:32:36.678925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:36.678895 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9be17b-23c3-446b-ae0b-ca419c143d1d" path="/var/lib/kubelet/pods/2d9be17b-23c3-446b-ae0b-ca419c143d1d/volumes" Nov 03 20:32:36.679386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:36.679369 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b72babb-158e-496a-bed3-3d6dfa3e1a6e" path="/var/lib/kubelet/pods/6b72babb-158e-496a-bed3-3d6dfa3e1a6e/volumes" Nov 03 20:32:36.679783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:36.679770 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4eced3-9286-4b3a-9d75-0fe80735f8b5" path="/var/lib/kubelet/pods/8f4eced3-9286-4b3a-9d75-0fe80735f8b5/volumes" Nov 03 20:32:36.942325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:36.942245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:37.884378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:37.884339 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:37.986650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:37.986601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:38.641034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:38.639522 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.028024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:40.027988 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.277505 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:40.277470 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.390443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:40.390404 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:41.470117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:41.468739 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm"] Nov 03 20:32:41.470117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:41.469011 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" podUID="0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" containerName="maas-api" containerID="cri-o://650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841" gracePeriod=30 Nov 03 20:32:41.527966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:41.527929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:41.730174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:41.730053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:41.846372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:41.845687 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.002141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:41.993824 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:32:42.057135 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.051554 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.063681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.063644 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\") pod \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.064233 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca\") pod \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.067243 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs\") pod \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.067325 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp\" (UniqueName: \"kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp\") pod \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\" (UID: \"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8\") " Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.068881 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca" pod "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" (UID: "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.071713 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.074304 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs" pod "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" (UID: "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:42.085101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.075489 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls" pod "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" (UID: "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:42.116113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.114619 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp" pod "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" (UID: "0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8"). InnerVolumeSpecName "kube-api-access-7t8kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:42.176114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.173542 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:42.176114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.173579 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:42.176114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.173595 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t8kp\" (UniqueName: \"kubernetes.io/projected/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8-kube-api-access-7t8kp\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:42.324524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.324492 5043 generic.go:356] "Generic (PLEG): container finished" podID="0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" containerID="650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841" exitCode=0 Nov 03 20:32:42.325250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.325224 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" event={"ID":"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8","Type":"ContainerDied","Data":"650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841"} Nov 03 20:32:42.325459 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.325440 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" event={"ID":"0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8","Type":"ContainerDied","Data":"309c02d0ba513225899133c1d2df071a36d15b6d7445ae7d507681af7c1c3faf"} Nov 03 20:32:42.325593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.325581 5043 scope.go:122] "RemoveContainer" containerID="650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841" Nov 03 20:32:42.325839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.325826 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm" Nov 03 20:32:42.342154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.342129 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.356235 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.356211 5043 scope.go:122] "RemoveContainer" containerID="650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841" Nov 03 20:32:42.356600 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:42.356573 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841\": container with ID starting with 650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841 not found: ID does not exist" containerID="650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841" Nov 03 20:32:42.356693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.356611 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841"} err="failed to get container status \"650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841\": rpc error: code = NotFound desc = could not find container \"650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841\": container with ID starting with 650589d8a16c3f170867deb97abed2f13020c3cef8409656e1e5e6e4f6049841 not found: ID does not exist" Nov 03 20:32:42.390471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.390435 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm"] Nov 03 20:32:42.401957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.401924 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fab871-7885c6967d-4w8lm"] Nov 03 20:32:42.451802 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.451766 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.552038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.551992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.684624 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.684552 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" path="/var/lib/kubelet/pods/0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8/volumes" Nov 03 20:32:42.950884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:42.950788 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:44.209324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:44.209281 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:46.204511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:46.204421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:47.822709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:47.822669 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:49.427552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:49.427521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:49.531024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:49.530986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:49.751942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:49.751851 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.267004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:51.266867 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.375021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:51.374986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.488043 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:51.488009 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.939934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:51.939892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.041853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:52.041811 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.958185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:52.958150 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9"] Nov 03 20:32:52.958589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:52.958404 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" podUID="c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" containerName="maas-api" containerID="cri-o://d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e" gracePeriod=30 Nov 03 20:32:53.046183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.046141 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:53.132993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.132957 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84"] Nov 03 20:32:53.161975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.161936 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v"] Nov 03 20:32:53.255962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.254987 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:53.294468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.294441 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:32:53.359339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.359308 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:53.425081 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.425031 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6"] Nov 03 20:32:53.425284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.425257 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" podUID="05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" containerName="maas-api" containerID="cri-o://70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2" gracePeriod=30 Nov 03 20:32:53.463729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.463005 5043 generic.go:356] "Generic (PLEG): container finished" podID="c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" containerID="d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e" exitCode=0 Nov 03 20:32:53.463729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.463215 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" event={"ID":"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd","Type":"ContainerDied","Data":"d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e"} Nov 03 20:32:53.463729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.463277 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" event={"ID":"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd","Type":"ContainerDied","Data":"339ff3d61b31d009d87bcacac860a2475f1d001270b7fa57d20af866e2b29a5c"} Nov 03 20:32:53.463729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.463311 5043 scope.go:122] "RemoveContainer" containerID="d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e" Nov 03 20:32:53.463729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.463496 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9" Nov 03 20:32:53.490058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.490020 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs\") pod \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " Nov 03 20:32:53.491141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.491109 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls\") pod \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " Nov 03 20:32:53.491264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.491232 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb\" (UniqueName: \"kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb\") pod \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " Nov 03 20:32:53.491327 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.491260 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca\") pod \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\" (UID: \"c39f3aa8-1aa5-4a26-a91c-e9b93187bffd\") " Nov 03 20:32:53.492144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.492058 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca" pod "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" (UID: "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:53.494749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.494729 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs" pod "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" (UID: "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:53.503313 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.503051 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:53.521857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.521782 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb" pod "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" (UID: "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd"). InnerVolumeSpecName "kube-api-access-5jskb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:53.523041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.523012 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls" pod "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" (UID: "c39f3aa8-1aa5-4a26-a91c-e9b93187bffd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:53.547011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.546939 5043 scope.go:122] "RemoveContainer" containerID="d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e" Nov 03 20:32:53.547470 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:53.547437 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e\": container with ID starting with d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e not found: ID does not exist" containerID="d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e" Nov 03 20:32:53.547584 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.547475 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e"} err="failed to get container status \"d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e\": rpc error: code = NotFound desc = could not find container \"d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e\": container with ID starting with d210ca91592ee48d50e3748d19e14f6d80a022d84af25c7ebb0dcb59a9f5c40e not found: ID does not exist" Nov 03 20:32:53.594311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.594275 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:53.594311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.594312 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:53.594560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.594326 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jskb\" (UniqueName: \"kubernetes.io/projected/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-kube-api-access-5jskb\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:53.594560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.594341 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:53.706717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.706692 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:32:53.907110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.906635 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs\") pod \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " Nov 03 20:32:53.907110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.906713 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r\" (UniqueName: \"kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r\") pod \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " Nov 03 20:32:53.907110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.906762 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca\") pod \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " Nov 03 20:32:53.907110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.906799 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls\") pod \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\" (UID: \"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02\") " Nov 03 20:32:53.930091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.916791 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca" pod "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" (UID: "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:53.930091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.923997 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs" pod "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" (UID: "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:53.943026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.942949 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls" pod "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" (UID: "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:53.971290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:53.971244 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r" pod "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" (UID: "05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02"). InnerVolumeSpecName "kube-api-access-bhr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:54.019725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.019153 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:54.019725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.019189 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhr6r\" (UniqueName: \"kubernetes.io/projected/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-kube-api-access-bhr6r\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:54.019725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.019206 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:54.019725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.019222 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:54.059260 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.059229 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9"] Nov 03 20:32:54.070678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.070637 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-17b3d9-8d99bf867-n2wz9"] Nov 03 20:32:54.411093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.411044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:54.480805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.480741 5043 generic.go:356] "Generic (PLEG): container finished" podID="05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" containerID="70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2" exitCode=0 Nov 03 20:32:54.480993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.480876 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" Nov 03 20:32:54.480993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.480953 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" event={"ID":"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02","Type":"ContainerDied","Data":"70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2"} Nov 03 20:32:54.481152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.481035 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6" event={"ID":"05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02","Type":"ContainerDied","Data":"1a48bc5761ffcbd09f11e40964229cf9213f899ceb4b288e8d5e3cd44de33a90"} Nov 03 20:32:54.481152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.481093 5043 scope.go:122] "RemoveContainer" containerID="70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2" Nov 03 20:32:54.493446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.493425 5043 scope.go:122] "RemoveContainer" containerID="70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2" Nov 03 20:32:54.493717 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:54.493696 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2\": container with ID starting with 70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2 not found: ID does not exist" containerID="70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2" Nov 03 20:32:54.493787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.493723 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2"} err="failed to get container status \"70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2\": rpc error: code = NotFound desc = could not find container \"70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2\": container with ID starting with 70b0e800058a7adebe292b8a45e674b2c9427eb241c40caf500aaa267aa6b1b2 not found: ID does not exist" Nov 03 20:32:54.515890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.515858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:54.534578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.534552 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6"] Nov 03 20:32:54.545620 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.545594 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1aa66-8544ddb67f-2pvx6"] Nov 03 20:32:54.615725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.615692 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:54.679810 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.679626 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" path="/var/lib/kubelet/pods/05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02/volumes" Nov 03 20:32:54.680699 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.680668 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" path="/var/lib/kubelet/pods/c39f3aa8-1aa5-4a26-a91c-e9b93187bffd/volumes" Nov 03 20:32:54.736210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.736181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:54.845515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:54.845477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:55.009705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:55.009599 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:55.335058 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:55.335021 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.286869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:56.286833 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:57.827275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:57.827231 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.133495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.133414 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" podUID="092a749e-6b8c-47b0-a314-5f1347057618" containerName="payload-processing" containerID="cri-o://6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432" gracePeriod=25 Nov 03 20:32:58.163144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.163113 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" podUID="b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" containerName="payload-pre-processing" containerID="cri-o://1695f1b6f6ded6d0b3bb2e07ac47c74a52fe119a2f2a850c9110ecf938f5bc5f" gracePeriod=25 Nov 03 20:32:58.478002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.477972 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:32:58.516631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.516600 5043 generic.go:356] "Generic (PLEG): container finished" podID="b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" containerID="1695f1b6f6ded6d0b3bb2e07ac47c74a52fe119a2f2a850c9110ecf938f5bc5f" exitCode=0 Nov 03 20:32:58.516863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.516835 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" event={"ID":"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb","Type":"ContainerDied","Data":"1695f1b6f6ded6d0b3bb2e07ac47c74a52fe119a2f2a850c9110ecf938f5bc5f"} Nov 03 20:32:58.517017 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.516897 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" event={"ID":"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb","Type":"ContainerDied","Data":"94923ad6aeb0070c0a1f666895a12e0e1d88872d69041fc7154414089ba3da06"} Nov 03 20:32:58.517017 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.516922 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94923ad6aeb0070c0a1f666895a12e0e1d88872d69041fc7154414089ba3da06" Nov 03 20:32:58.518640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.518615 5043 generic.go:356] "Generic (PLEG): container finished" podID="092a749e-6b8c-47b0-a314-5f1347057618" containerID="6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432" exitCode=0 Nov 03 20:32:58.518755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.518668 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" Nov 03 20:32:58.518755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.518676 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" event={"ID":"092a749e-6b8c-47b0-a314-5f1347057618","Type":"ContainerDied","Data":"6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432"} Nov 03 20:32:58.518755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.518711 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84" event={"ID":"092a749e-6b8c-47b0-a314-5f1347057618","Type":"ContainerDied","Data":"726043e993bcd6bf60ba847474ba714b2d7f6fd47e4e911507b4bb084adedfe2"} Nov 03 20:32:58.518755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.518739 5043 scope.go:122] "RemoveContainer" containerID="6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432" Nov 03 20:32:58.527447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.527423 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:32:58.533184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.533163 5043 scope.go:122] "RemoveContainer" containerID="6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432" Nov 03 20:32:58.533515 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:32:58.533488 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432\": container with ID starting with 6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432 not found: ID does not exist" containerID="6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432" Nov 03 20:32:58.533614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.533514 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432"} err="failed to get container status \"6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432\": rpc error: code = NotFound desc = could not find container \"6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432\": container with ID starting with 6e80067cfef22faa7d8d38f33bc6fe9ccda6310cb94b99069a369e5803cce432 not found: ID does not exist" Nov 03 20:32:58.582918 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.582886 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume\") pod \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " Nov 03 20:32:58.583056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.583013 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz\" (UniqueName: \"kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz\") pod \"092a749e-6b8c-47b0-a314-5f1347057618\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " Nov 03 20:32:58.583294 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.583275 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume\") pod \"092a749e-6b8c-47b0-a314-5f1347057618\" (UID: \"092a749e-6b8c-47b0-a314-5f1347057618\") " Nov 03 20:32:58.583399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.583383 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8\" (UniqueName: \"kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8\") pod \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\" (UID: \"b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb\") " Nov 03 20:32:58.584588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.584537 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume" pod "092a749e-6b8c-47b0-a314-5f1347057618" (UID: "092a749e-6b8c-47b0-a314-5f1347057618"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.584588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.584564 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume" pod "b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" (UID: "b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.586533 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.586507 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8" pod "b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" (UID: "b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb"). InnerVolumeSpecName "kube-api-access-jc4m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.586653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.586592 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz" pod "092a749e-6b8c-47b0-a314-5f1347057618" (UID: "092a749e-6b8c-47b0-a314-5f1347057618"). InnerVolumeSpecName "kube-api-access-f67kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.692813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.692734 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092a749e-6b8c-47b0-a314-5f1347057618-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:58.693023 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.692999 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4m8\" (UniqueName: \"kubernetes.io/projected/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-kube-api-access-jc4m8\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:58.693194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.693026 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:58.693194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.693042 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67kz\" (UniqueName: \"kubernetes.io/projected/092a749e-6b8c-47b0-a314-5f1347057618-kube-api-access-f67kz\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:32:58.863796 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.863745 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84"] Nov 03 20:32:58.868201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:58.868175 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-17b3d9-7f778877db-7jq84"] Nov 03 20:32:59.418293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:59.418203 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:59.530906 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:59.530875 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v" Nov 03 20:32:59.589458 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:59.589426 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v"] Nov 03 20:32:59.599198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:32:59.599174 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-17b3d9-766cf874bc-t568v"] Nov 03 20:33:00.680007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:00.679971 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092a749e-6b8c-47b0-a314-5f1347057618" path="/var/lib/kubelet/pods/092a749e-6b8c-47b0-a314-5f1347057618/volumes" Nov 03 20:33:00.680663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:00.680643 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" path="/var/lib/kubelet/pods/b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb/volumes" Nov 03 20:33:01.021451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:01.021360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:01.163221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:01.163181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:01.331428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:01.331395 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.497629 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:02.497591 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.623155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:02.623122 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.751724 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:02.751641 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.886650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:02.886617 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.062730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:03.062507 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.189965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:03.189932 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.318545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:03.318464 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.773722 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:03.773687 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.254698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:04.254665 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.392644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:04.392609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.907884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:04.907849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.634020 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:05.633986 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:33:06.710198 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:06.710153 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.119835 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:08.119799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.222407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:08.222336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.438816 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:08.438726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.622730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:08.622690 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:10.019780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:10.019730 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:10.122642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:10.122592 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:10.689716 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:10.689672 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.054448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.054414 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp"] Nov 03 20:33:12.054932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.054605 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" podUID="277c75ae-0568-4c6c-bcfb-b574312c43ca" containerName="maas-api" containerID="cri-o://63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa" gracePeriod=30 Nov 03 20:33:12.135907 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.135831 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.391573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.391536 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.400042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.399987 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:33:12.549753 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.549715 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k\" (UniqueName: \"kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k\") pod \"277c75ae-0568-4c6c-bcfb-b574312c43ca\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " Nov 03 20:33:12.549959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.549870 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs\") pod \"277c75ae-0568-4c6c-bcfb-b574312c43ca\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " Nov 03 20:33:12.549959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.549936 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls\") pod \"277c75ae-0568-4c6c-bcfb-b574312c43ca\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " Nov 03 20:33:12.550207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.550180 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca\") pod \"277c75ae-0568-4c6c-bcfb-b574312c43ca\" (UID: \"277c75ae-0568-4c6c-bcfb-b574312c43ca\") " Nov 03 20:33:12.551270 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.551235 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca" pod "277c75ae-0568-4c6c-bcfb-b574312c43ca" (UID: "277c75ae-0568-4c6c-bcfb-b574312c43ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:12.552610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.552588 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls" pod "277c75ae-0568-4c6c-bcfb-b574312c43ca" (UID: "277c75ae-0568-4c6c-bcfb-b574312c43ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:12.552903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.552875 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k" pod "277c75ae-0568-4c6c-bcfb-b574312c43ca" (UID: "277c75ae-0568-4c6c-bcfb-b574312c43ca"). InnerVolumeSpecName "kube-api-access-l4w4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.553157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.553132 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs" pod "277c75ae-0568-4c6c-bcfb-b574312c43ca" (UID: "277c75ae-0568-4c6c-bcfb-b574312c43ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:12.623360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.623030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.653652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.653610 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:12.653911 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.653889 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/277c75ae-0568-4c6c-bcfb-b574312c43ca-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:12.654218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.653916 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/277c75ae-0568-4c6c-bcfb-b574312c43ca-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:12.654218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.653931 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4w4k\" (UniqueName: \"kubernetes.io/projected/277c75ae-0568-4c6c-bcfb-b574312c43ca-kube-api-access-l4w4k\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:12.723895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.710326 5043 generic.go:356] "Generic (PLEG): container finished" podID="277c75ae-0568-4c6c-bcfb-b574312c43ca" containerID="63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa" exitCode=0 Nov 03 20:33:12.723895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.710528 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" Nov 03 20:33:12.731271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.730910 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.734175 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.734151 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" event={"ID":"277c75ae-0568-4c6c-bcfb-b574312c43ca","Type":"ContainerDied","Data":"63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa"} Nov 03 20:33:12.734350 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.734333 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp" event={"ID":"277c75ae-0568-4c6c-bcfb-b574312c43ca","Type":"ContainerDied","Data":"6c1713e1151f101402967817061f0f37a10723a4ffdab91f4e76bd0ecfa4d3b0"} Nov 03 20:33:12.734442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.734429 5043 scope.go:122] "RemoveContainer" containerID="63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa" Nov 03 20:33:12.752540 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.752504 5043 scope.go:122] "RemoveContainer" containerID="63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa" Nov 03 20:33:12.752843 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:12.752818 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa\": container with ID starting with 63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa not found: ID does not exist" containerID="63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa" Nov 03 20:33:12.752933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.752851 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa"} err="failed to get container status \"63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa\": rpc error: code = NotFound desc = could not find container \"63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa\": container with ID starting with 63864b64e78bbff1ba151bc25972bfec0e143004acba39dc9d5e90fa4077a0aa not found: ID does not exist" Nov 03 20:33:12.767158 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.767134 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp"] Nov 03 20:33:12.780152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.778409 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7e4389-c69667cfb-djbdp"] Nov 03 20:33:12.828837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:12.826026 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.234252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:13.234214 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.160729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.160697 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld"] Nov 03 20:33:14.160959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.160933 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" podUID="10969fd1-504d-4fc7-8236-73fde6043259" containerName="maas-api" containerID="cri-o://a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9" gracePeriod=30 Nov 03 20:33:14.225092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.224376 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.227408 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.226811 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj"] Nov 03 20:33:14.235772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.235748 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr"] Nov 03 20:33:14.363822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.363783 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.455932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.455854 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.506695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.506669 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:33:14.680067 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.680029 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277c75ae-0568-4c6c-bcfb-b574312c43ca" path="/var/lib/kubelet/pods/277c75ae-0568-4c6c-bcfb-b574312c43ca/volumes" Nov 03 20:33:14.686006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.685978 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs\") pod \"10969fd1-504d-4fc7-8236-73fde6043259\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " Nov 03 20:33:14.686187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.686037 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj\" (UniqueName: \"kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj\") pod \"10969fd1-504d-4fc7-8236-73fde6043259\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " Nov 03 20:33:14.686187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.686121 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca\") pod \"10969fd1-504d-4fc7-8236-73fde6043259\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " Nov 03 20:33:14.686187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.686159 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls\") pod \"10969fd1-504d-4fc7-8236-73fde6043259\" (UID: \"10969fd1-504d-4fc7-8236-73fde6043259\") " Nov 03 20:33:14.686538 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.686496 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca" pod "10969fd1-504d-4fc7-8236-73fde6043259" (UID: "10969fd1-504d-4fc7-8236-73fde6043259"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:14.686817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.686794 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10969fd1-504d-4fc7-8236-73fde6043259-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:14.687912 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.687885 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.688580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.688554 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls" pod "10969fd1-504d-4fc7-8236-73fde6043259" (UID: "10969fd1-504d-4fc7-8236-73fde6043259"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:14.688681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.688590 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs" pod "10969fd1-504d-4fc7-8236-73fde6043259" (UID: "10969fd1-504d-4fc7-8236-73fde6043259"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:14.688725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.688685 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj" pod "10969fd1-504d-4fc7-8236-73fde6043259" (UID: "10969fd1-504d-4fc7-8236-73fde6043259"). InnerVolumeSpecName "kube-api-access-dq2fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:14.730152 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.730064 5043 generic.go:356] "Generic (PLEG): container finished" podID="10969fd1-504d-4fc7-8236-73fde6043259" containerID="a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9" exitCode=0 Nov 03 20:33:14.730281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.730155 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" Nov 03 20:33:14.730281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.730166 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" event={"ID":"10969fd1-504d-4fc7-8236-73fde6043259","Type":"ContainerDied","Data":"a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9"} Nov 03 20:33:14.730281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.730221 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld" event={"ID":"10969fd1-504d-4fc7-8236-73fde6043259","Type":"ContainerDied","Data":"fc6291ccf805e751d21a823b270a18cd997fe2a42d196fb095e01e3b5fb79ee0"} Nov 03 20:33:14.730281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.730246 5043 scope.go:122] "RemoveContainer" containerID="a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9" Nov 03 20:33:14.743006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.742962 5043 scope.go:122] "RemoveContainer" containerID="a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9" Nov 03 20:33:14.743344 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:14.743321 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9\": container with ID starting with a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9 not found: ID does not exist" containerID="a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9" Nov 03 20:33:14.743453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.743354 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9"} err="failed to get container status \"a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9\": rpc error: code = NotFound desc = could not find container \"a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9\": container with ID starting with a31d02366a7cc142939b58e4a4d82549ce6fbf03db47b84fc601890f45e856e9 not found: ID does not exist" Nov 03 20:33:14.784510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.784482 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld"] Nov 03 20:33:14.788573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.788544 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:14.788707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.788576 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2fj\" (UniqueName: \"kubernetes.io/projected/10969fd1-504d-4fc7-8236-73fde6043259-kube-api-access-dq2fj\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:14.788707 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.788592 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10969fd1-504d-4fc7-8236-73fde6043259-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:14.790728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.790701 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.790838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.790712 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-e3ba87-598b488f56-lkgld"] Nov 03 20:33:14.877379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:14.877339 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:15.265982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:15.265945 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.646565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:16.646528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.695150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:16.695110 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10969fd1-504d-4fc7-8236-73fde6043259" path="/var/lib/kubelet/pods/10969fd1-504d-4fc7-8236-73fde6043259/volumes" Nov 03 20:33:18.358047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:18.358010 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.227794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.227753 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" podUID="a64147b6-3d77-485d-979f-09b3cdbdfe49" containerName="payload-processing" containerID="cri-o://36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb" gracePeriod=25 Nov 03 20:33:19.237349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.237317 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" podUID="bafe64f1-b7d6-4819-ac96-1d6851b30dde" containerName="payload-pre-processing" containerID="cri-o://848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe" gracePeriod=25 Nov 03 20:33:19.291183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.291139 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.395088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.394864 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.599182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.597751 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:33:19.641726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.641698 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:33:19.671041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671008 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume\") pod \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " Nov 03 20:33:19.671233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671061 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc\" (UniqueName: \"kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc\") pod \"a64147b6-3d77-485d-979f-09b3cdbdfe49\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " Nov 03 20:33:19.671233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671136 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47\" (UniqueName: \"kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47\") pod \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\" (UID: \"bafe64f1-b7d6-4819-ac96-1d6851b30dde\") " Nov 03 20:33:19.671376 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671271 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume\") pod \"a64147b6-3d77-485d-979f-09b3cdbdfe49\" (UID: \"a64147b6-3d77-485d-979f-09b3cdbdfe49\") " Nov 03 20:33:19.671515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671483 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume" pod "bafe64f1-b7d6-4819-ac96-1d6851b30dde" (UID: "bafe64f1-b7d6-4819-ac96-1d6851b30dde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:19.671645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671591 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume" pod "a64147b6-3d77-485d-979f-09b3cdbdfe49" (UID: "a64147b6-3d77-485d-979f-09b3cdbdfe49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:19.671886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671867 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe64f1-b7d6-4819-ac96-1d6851b30dde-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:19.671953 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.671890 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a64147b6-3d77-485d-979f-09b3cdbdfe49-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:19.673244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.673218 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47" pod "bafe64f1-b7d6-4819-ac96-1d6851b30dde" (UID: "bafe64f1-b7d6-4819-ac96-1d6851b30dde"). InnerVolumeSpecName "kube-api-access-bks47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:19.673523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.673502 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc" pod "a64147b6-3d77-485d-979f-09b3cdbdfe49" (UID: "a64147b6-3d77-485d-979f-09b3cdbdfe49"). InnerVolumeSpecName "kube-api-access-6vcnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:19.717938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.717890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.908571 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vcnc\" (UniqueName: \"kubernetes.io/projected/a64147b6-3d77-485d-979f-09b3cdbdfe49-kube-api-access-6vcnc\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.908606 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bks47\" (UniqueName: \"kubernetes.io/projected/bafe64f1-b7d6-4819-ac96-1d6851b30dde-kube-api-access-bks47\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.915576 5043 generic.go:356] "Generic (PLEG): container finished" podID="a64147b6-3d77-485d-979f-09b3cdbdfe49" containerID="36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb" exitCode=0 Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.915788 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" event={"ID":"a64147b6-3d77-485d-979f-09b3cdbdfe49","Type":"ContainerDied","Data":"36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb"} Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.915844 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" event={"ID":"a64147b6-3d77-485d-979f-09b3cdbdfe49","Type":"ContainerDied","Data":"4139b3f83f80b0505de5995bfa6bce9aaf31f37c14f33ecbf182868252f47874"} Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.915874 5043 scope.go:122] "RemoveContainer" containerID="36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb" Nov 03 20:33:19.920035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.916003 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj" Nov 03 20:33:19.929405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.929327 5043 generic.go:356] "Generic (PLEG): container finished" podID="bafe64f1-b7d6-4819-ac96-1d6851b30dde" containerID="848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe" exitCode=0 Nov 03 20:33:19.929405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.929380 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" event={"ID":"bafe64f1-b7d6-4819-ac96-1d6851b30dde","Type":"ContainerDied","Data":"848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe"} Nov 03 20:33:19.929628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.929444 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" event={"ID":"bafe64f1-b7d6-4819-ac96-1d6851b30dde","Type":"ContainerDied","Data":"0786ac20435ce5a002850e7e9915e182c4b9bdede02d59cdf34db66990db04c7"} Nov 03 20:33:19.929628 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.929488 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr" Nov 03 20:33:19.947616 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.947590 5043 scope.go:122] "RemoveContainer" containerID="36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb" Nov 03 20:33:19.947941 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:19.947917 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb\": container with ID starting with 36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb not found: ID does not exist" containerID="36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb" Nov 03 20:33:19.948003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.947955 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb"} err="failed to get container status \"36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb\": rpc error: code = NotFound desc = could not find container \"36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb\": container with ID starting with 36e0ab47b0da7841fa7ebd60183f2e5aa6a380176a31aa5c72cb6bd31269befb not found: ID does not exist" Nov 03 20:33:19.948003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.947979 5043 scope.go:122] "RemoveContainer" containerID="848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe" Nov 03 20:33:19.963316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.963291 5043 scope.go:122] "RemoveContainer" containerID="848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe" Nov 03 20:33:19.963642 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:19.963620 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe\": container with ID starting with 848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe not found: ID does not exist" containerID="848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe" Nov 03 20:33:19.963729 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.963653 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe"} err="failed to get container status \"848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe\": rpc error: code = NotFound desc = could not find container \"848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe\": container with ID starting with 848ab45652512432554caefadc5e38503cad0da110536a8a0403888b0fde03fe not found: ID does not exist" Nov 03 20:33:19.988811 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.988776 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj"] Nov 03 20:33:19.992469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:19.992445 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-9e1bcb-9f4dccdbb-qbpsj"] Nov 03 20:33:20.022588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.022558 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr"] Nov 03 20:33:20.027564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.027543 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-9e1bcb-598c6f96ff-vsvdr"] Nov 03 20:33:20.052275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.052245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.209577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.209495 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.393491 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.393453 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.679395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.679354 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64147b6-3d77-485d-979f-09b3cdbdfe49" path="/var/lib/kubelet/pods/a64147b6-3d77-485d-979f-09b3cdbdfe49/volumes" Nov 03 20:33:20.679930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:20.679908 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafe64f1-b7d6-4819-ac96-1d6851b30dde" path="/var/lib/kubelet/pods/bafe64f1-b7d6-4819-ac96-1d6851b30dde/volumes" Nov 03 20:33:22.022348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.022312 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.131749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.131706 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.261517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.261481 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.428166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.428132 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt"] Nov 03 20:33:22.428370 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.428340 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" containerID="cri-o://674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926" gracePeriod=30 Nov 03 20:33:22.461112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.461082 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.899377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:22.899338 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:23.914151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:23.913906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:24.023909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:24.023877 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:25.127394 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:25.127356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:25.949550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:25.949512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:27.588197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.587968 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:27.802652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.802621 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:33:27.869822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.869788 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr"] Nov 03 20:33:27.939507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939471 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.939693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939540 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.939693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939571 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.939693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939655 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.939852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939798 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.939917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939858 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.939971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939914 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.940027 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.939989 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.940128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.940013 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:27.940128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.940055 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:27.940128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.940095 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo\") pod \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\" (UID: \"2eff5c5d-7869-41ea-b7fe-f499411cec5a\") " Nov 03 20:33:27.940599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.940089 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:27.940599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.940253 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:27.940599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.940294 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:27.948143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.946639 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:27.948143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.946942 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "kube-api-access-g8qsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948758 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948788 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948802 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948822 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948836 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948848 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eff5c5d-7869-41ea-b7fe-f499411cec5a-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.948862 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qsl\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-kube-api-access-g8qsl\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:27.952188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.951186 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:27.958803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.958389 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token" pod "2eff5c5d-7869-41ea-b7fe-f499411cec5a" (UID: "2eff5c5d-7869-41ea-b7fe-f499411cec5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:27.981367 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:27.981341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:28.014422 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.014330 5043 generic.go:356] "Generic (PLEG): container finished" podID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerID="674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926" exitCode=0 Nov 03 20:33:28.014422 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.014408 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" Nov 03 20:33:28.014604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.014413 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" event={"ID":"2eff5c5d-7869-41ea-b7fe-f499411cec5a","Type":"ContainerDied","Data":"674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926"} Nov 03 20:33:28.014604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.014518 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt" event={"ID":"2eff5c5d-7869-41ea-b7fe-f499411cec5a","Type":"ContainerDied","Data":"c1ac5ad0c25d3ecb6ba5effdf11edeeae59f6251774b28d8e15ed51463184039"} Nov 03 20:33:28.014604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.014549 5043 scope.go:122] "RemoveContainer" containerID="674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926" Nov 03 20:33:28.032558 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.032541 5043 scope.go:122] "RemoveContainer" containerID="674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926" Nov 03 20:33:28.032867 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:28.032848 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926\": container with ID starting with 674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926 not found: ID does not exist" containerID="674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926" Nov 03 20:33:28.032919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.032877 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926"} err="failed to get container status \"674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926\": rpc error: code = NotFound desc = could not find container \"674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926\": container with ID starting with 674c2866b32ad0c043f1f66cd914c3bd7411a46b310abcce716a5c1d51696926 not found: ID does not exist" Nov 03 20:33:28.049768 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.049744 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:28.049873 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.049770 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eff5c5d-7869-41ea-b7fe-f499411cec5a-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:28.071852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.071826 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt"] Nov 03 20:33:28.075461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.075440 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-7h8dt"] Nov 03 20:33:28.679376 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:28.679342 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" path="/var/lib/kubelet/pods/2eff5c5d-7869-41ea-b7fe-f499411cec5a/volumes" Nov 03 20:33:29.247801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:29.247767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:30.961608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:30.961571 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:31.221728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:31.221644 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:31.335368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:31.335336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:31.638030 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:31.637995 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg"] Nov 03 20:33:31.673188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:31.673156 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn"] Nov 03 20:33:31.790469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:31.790439 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:31.889267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:31.889170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.093932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:32.093727 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.832193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:32.832154 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.757703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:33.757671 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th"] Nov 03 20:33:33.795479 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:33.795449 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk"] Nov 03 20:33:33.905102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:33.901237 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.009446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:34.009219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.514789 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:34.514752 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:35.630289 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:35.630249 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:33:36.162668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:36.162632 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:36.638366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:36.638327 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" podUID="53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" containerName="payload-processing" containerID="cri-o://ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587" gracePeriod=25 Nov 03 20:33:36.673691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:36.673657 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" podUID="c6eae762-603f-471d-8b1a-67fdb9fb6db8" containerName="payload-pre-processing" containerID="cri-o://320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d" gracePeriod=25 Nov 03 20:33:36.988372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:36.988344 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:33:37.032661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.032638 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:33:37.084685 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.084646 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume\") pod \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " Nov 03 20:33:37.085147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.085116 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume" pod "53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" (UID: "53011f9e-2a42-4c14-a8e5-f8c45d12bf3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:37.085147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.084697 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume\") pod \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " Nov 03 20:33:37.085347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.085146 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume" pod "c6eae762-603f-471d-8b1a-67fdb9fb6db8" (UID: "c6eae762-603f-471d-8b1a-67fdb9fb6db8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:37.085347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.085224 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw\" (UniqueName: \"kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw\") pod \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\" (UID: \"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c\") " Nov 03 20:33:37.085347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.085271 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp\" (UniqueName: \"kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp\") pod \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\" (UID: \"c6eae762-603f-471d-8b1a-67fdb9fb6db8\") " Nov 03 20:33:37.085930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.085908 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6eae762-603f-471d-8b1a-67fdb9fb6db8-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:37.086040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.085935 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:37.088243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.088217 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp" pod "c6eae762-603f-471d-8b1a-67fdb9fb6db8" (UID: "c6eae762-603f-471d-8b1a-67fdb9fb6db8"). InnerVolumeSpecName "kube-api-access-zp4mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:37.088364 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.088220 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw" pod "53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" (UID: "53011f9e-2a42-4c14-a8e5-f8c45d12bf3c"). InnerVolumeSpecName "kube-api-access-7mwlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:37.115458 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.115428 5043 generic.go:356] "Generic (PLEG): container finished" podID="53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" containerID="ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587" exitCode=0 Nov 03 20:33:37.115612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.115595 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" event={"ID":"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c","Type":"ContainerDied","Data":"ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587"} Nov 03 20:33:37.115719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.115647 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" event={"ID":"53011f9e-2a42-4c14-a8e5-f8c45d12bf3c","Type":"ContainerDied","Data":"23b24ec6a1aff7ea9458dc528b8d7b21138a0ea6e1a5379ca8ad2ede97ea3725"} Nov 03 20:33:37.115719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.115675 5043 scope.go:122] "RemoveContainer" containerID="ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587" Nov 03 20:33:37.115842 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.115812 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg" Nov 03 20:33:37.127896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.127871 5043 generic.go:356] "Generic (PLEG): container finished" podID="c6eae762-603f-471d-8b1a-67fdb9fb6db8" containerID="320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d" exitCode=0 Nov 03 20:33:37.128478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.128033 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" event={"ID":"c6eae762-603f-471d-8b1a-67fdb9fb6db8","Type":"ContainerDied","Data":"320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d"} Nov 03 20:33:37.128478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.128296 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" Nov 03 20:33:37.130187 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.130156 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn" event={"ID":"c6eae762-603f-471d-8b1a-67fdb9fb6db8","Type":"ContainerDied","Data":"e8f4c7d5bf3b3f2d6f72e845e8b614c262b703f59df96c51f89b90190fe5cdd0"} Nov 03 20:33:37.141447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.141429 5043 scope.go:122] "RemoveContainer" containerID="ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587" Nov 03 20:33:37.141749 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:37.141730 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587\": container with ID starting with ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587 not found: ID does not exist" containerID="ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587" Nov 03 20:33:37.141826 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.141759 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587"} err="failed to get container status \"ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587\": rpc error: code = NotFound desc = could not find container \"ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587\": container with ID starting with ee602bd51a4494047ae9c329254bb003c868c2e39b56c8922ed482ed6a238587 not found: ID does not exist" Nov 03 20:33:37.141826 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.141781 5043 scope.go:122] "RemoveContainer" containerID="320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d" Nov 03 20:33:37.153477 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.153457 5043 scope.go:122] "RemoveContainer" containerID="320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d" Nov 03 20:33:37.153763 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:37.153744 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d\": container with ID starting with 320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d not found: ID does not exist" containerID="320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d" Nov 03 20:33:37.153834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.153767 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d"} err="failed to get container status \"320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d\": rpc error: code = NotFound desc = could not find container \"320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d\": container with ID starting with 320229908fed91afc06d43f2ccd2ff3724cbedd0cb923b851c58103ea970a90d not found: ID does not exist" Nov 03 20:33:37.187181 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.187123 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwlw\" (UniqueName: \"kubernetes.io/projected/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c-kube-api-access-7mwlw\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:37.187181 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.187148 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4mp\" (UniqueName: \"kubernetes.io/projected/c6eae762-603f-471d-8b1a-67fdb9fb6db8-kube-api-access-zp4mp\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:37.188804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.188779 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg"] Nov 03 20:33:37.193249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.193226 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9973d9-f56bdd84d-fftbg"] Nov 03 20:33:37.201014 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:37.200981 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53011f9e_2a42_4c14_a8e5_f8c45d12bf3c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eae762_603f_471d_8b1a_67fdb9fb6db8.slice/crio-e8f4c7d5bf3b3f2d6f72e845e8b614c262b703f59df96c51f89b90190fe5cdd0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eae762_603f_471d_8b1a_67fdb9fb6db8.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:37.226713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.226685 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn"] Nov 03 20:33:37.230888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.230866 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9973d9-776bccfd95-hwqmn"] Nov 03 20:33:37.793372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:37.793331 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:38.679274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:38.679240 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" path="/var/lib/kubelet/pods/53011f9e-2a42-4c14-a8e5-f8c45d12bf3c/volumes" Nov 03 20:33:38.679889 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:38.679868 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6eae762-603f-471d-8b1a-67fdb9fb6db8" path="/var/lib/kubelet/pods/c6eae762-603f-471d-8b1a-67fdb9fb6db8/volumes" Nov 03 20:33:38.758691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:38.758636 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerName="payload-processing" containerID="cri-o://c276c09521edb817935aea7eadd0e4345a4b4c8183c209328fd0b9d9944ec001" gracePeriod=25 Nov 03 20:33:38.796508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:38.796443 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerName="payload-pre-processing" containerID="cri-o://b31767f6f593d6cf34b2e64cff7b7ff60df272dc5360a9748aed242e330bb511" gracePeriod=25 Nov 03 20:33:39.168448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.168411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:39.173301 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.173272 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:33:39.174441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.174419 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:33:39.185609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.185587 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:33:39.187267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.187245 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:33:39.393823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.393780 5043 scope.go:122] "RemoveContainer" containerID="1695f1b6f6ded6d0b3bb2e07ac47c74a52fe119a2f2a850c9110ecf938f5bc5f" Nov 03 20:33:39.690936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.690840 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:39.794673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.794634 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:39.900096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:39.896868 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.290590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:40.290552 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.840717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:40.838972 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:41.742615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:41.742578 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.900369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:42.900287 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" containerID="cri-o://b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238" gracePeriod=45 Nov 03 20:33:43.190433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.190403 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:33:43.215698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.215667 5043 generic.go:356] "Generic (PLEG): container finished" podID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerID="b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238" exitCode=0 Nov 03 20:33:43.215857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.215708 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" Nov 03 20:33:43.215857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.215744 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" event={"ID":"334cc1ee-0aa4-451f-b1e8-0be38daaff49","Type":"ContainerDied","Data":"b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238"} Nov 03 20:33:43.215857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.215785 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr" event={"ID":"334cc1ee-0aa4-451f-b1e8-0be38daaff49","Type":"ContainerDied","Data":"61f88290407eeae277f5c43e8bde1018f81ed1f94d6e1893a425af75dfd998bb"} Nov 03 20:33:43.215857 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.215808 5043 scope.go:122] "RemoveContainer" containerID="b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238" Nov 03 20:33:43.232568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.232549 5043 scope.go:122] "RemoveContainer" containerID="b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238" Nov 03 20:33:43.233160 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:33:43.233051 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238\": container with ID starting with b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238 not found: ID does not exist" containerID="b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238" Nov 03 20:33:43.233290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.233164 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238"} err="failed to get container status \"b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238\": rpc error: code = NotFound desc = could not find container \"b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238\": container with ID starting with b5a4a3e8c94fc505a9d9b0391969a40411c3e885d5795fb25d4cbe4213799238 not found: ID does not exist" Nov 03 20:33:43.289374 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.289335 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:43.299705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.298392 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home\") pod \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " Nov 03 20:33:43.299705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.298532 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g\" (UniqueName: \"kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g\") pod \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " Nov 03 20:33:43.299705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.298682 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm\") pod \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " Nov 03 20:33:43.299705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.298732 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir\") pod \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " Nov 03 20:33:43.299705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.298795 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache\") pod \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " Nov 03 20:33:43.299705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.298838 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs\" (UniqueName: \"kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs\") pod \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\" (UID: \"334cc1ee-0aa4-451f-b1e8-0be38daaff49\") " Nov 03 20:33:43.300418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.300056 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home" pod "334cc1ee-0aa4-451f-b1e8-0be38daaff49" (UID: "334cc1ee-0aa4-451f-b1e8-0be38daaff49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:43.300418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.300364 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir" pod "334cc1ee-0aa4-451f-b1e8-0be38daaff49" (UID: "334cc1ee-0aa4-451f-b1e8-0be38daaff49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:43.300798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.300757 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache" pod "334cc1ee-0aa4-451f-b1e8-0be38daaff49" (UID: "334cc1ee-0aa4-451f-b1e8-0be38daaff49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:43.303904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.303423 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs" pod "334cc1ee-0aa4-451f-b1e8-0be38daaff49" (UID: "334cc1ee-0aa4-451f-b1e8-0be38daaff49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:43.306124 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.306035 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm" pod "334cc1ee-0aa4-451f-b1e8-0be38daaff49" (UID: "334cc1ee-0aa4-451f-b1e8-0be38daaff49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:43.313531 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.310156 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g" pod "334cc1ee-0aa4-451f-b1e8-0be38daaff49" (UID: "334cc1ee-0aa4-451f-b1e8-0be38daaff49"). InnerVolumeSpecName "kube-api-access-rz87g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:43.401676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.401635 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-model-cache\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:43.401676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.401674 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tls-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:43.401676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.401687 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-home\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:43.401985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.401701 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz87g\" (UniqueName: \"kubernetes.io/projected/334cc1ee-0aa4-451f-b1e8-0be38daaff49-kube-api-access-rz87g\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:43.401985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.401715 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-dshm\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:43.401985 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.401729 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334cc1ee-0aa4-451f-b1e8-0be38daaff49-tmp-dir\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:43.473266 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.473192 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:43.585451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.585414 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr"] Nov 03 20:33:43.590937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:43.590907 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-w5-fd01ad-kserve-84b585bc4d-2gqzr"] Nov 03 20:33:44.300565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:44.300533 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:44.677832 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:44.677796 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" path="/var/lib/kubelet/pods/334cc1ee-0aa4-451f-b1e8-0be38daaff49/volumes" Nov 03 20:33:45.046145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:45.044830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.152430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:45.152334 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.347203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:45.347171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:47.023758 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:47.023717 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:47.582237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:47.582202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:48.780231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:48.780170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:48.890653 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:48.890583 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.170633 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:49.170600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:50.521734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:50.521687 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:52.207506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:52.207468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:53.290634 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:53.290596 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:53.608794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:53.608743 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:53.711636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:53.711578 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.038813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.038708 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.146724 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.146681 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.156162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.156124 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s"] Nov 03 20:33:54.277764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.277705 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.300822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.300735 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:54.650091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.645571 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.987979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:54.987895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:55.651066 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:55.651027 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:56.421430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:56.421389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:56.518297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:56.518261 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:57.015883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:57.015844 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:58.382736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:58.382700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:58.515513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:58.515470 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:58.620118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:58.619736 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:59.365192 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.365159 5043 generic.go:356] "Generic (PLEG): container finished" podID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerID="b31767f6f593d6cf34b2e64cff7b7ff60df272dc5360a9748aed242e330bb511" exitCode=0 Nov 03 20:33:59.365382 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.365358 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" event={"ID":"c367b1f5-858e-4c27-bcfc-56af079f456b","Type":"ContainerDied","Data":"b31767f6f593d6cf34b2e64cff7b7ff60df272dc5360a9748aed242e330bb511"} Nov 03 20:33:59.365498 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.365478 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" event={"ID":"c367b1f5-858e-4c27-bcfc-56af079f456b","Type":"ContainerDied","Data":"748ce6c724a3d7b204ebd29f1f3d450ca13bd4cc3ea85e72cefaf75866a9b8f7"} Nov 03 20:33:59.365587 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.365518 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748ce6c724a3d7b204ebd29f1f3d450ca13bd4cc3ea85e72cefaf75866a9b8f7" Nov 03 20:33:59.367378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.367354 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:33:59.367935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.367909 5043 generic.go:356] "Generic (PLEG): container finished" podID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerID="c276c09521edb817935aea7eadd0e4345a4b4c8183c209328fd0b9d9944ec001" exitCode=0 Nov 03 20:33:59.368043 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.367979 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" event={"ID":"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c","Type":"ContainerDied","Data":"c276c09521edb817935aea7eadd0e4345a4b4c8183c209328fd0b9d9944ec001"} Nov 03 20:33:59.406549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.406524 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:33:59.444957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.444931 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42\" (UniqueName: \"kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42\") pod \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " Nov 03 20:33:59.445120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.445015 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume\") pod \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\" (UID: \"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c\") " Nov 03 20:33:59.445195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.445162 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497\" (UniqueName: \"kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497\") pod \"c367b1f5-858e-4c27-bcfc-56af079f456b\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " Nov 03 20:33:59.445347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.445297 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume\") pod \"c367b1f5-858e-4c27-bcfc-56af079f456b\" (UID: \"c367b1f5-858e-4c27-bcfc-56af079f456b\") " Nov 03 20:33:59.445506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.445476 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume" pod "f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" (UID: "f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:59.445835 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.445812 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:59.445944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.445896 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume" pod "c367b1f5-858e-4c27-bcfc-56af079f456b" (UID: "c367b1f5-858e-4c27-bcfc-56af079f456b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:59.447371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.447346 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497" pod "c367b1f5-858e-4c27-bcfc-56af079f456b" (UID: "c367b1f5-858e-4c27-bcfc-56af079f456b"). InnerVolumeSpecName "kube-api-access-v9497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:59.447596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.447577 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42" pod "f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" (UID: "f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c"). InnerVolumeSpecName "kube-api-access-zkh42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:59.549414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.549372 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9497\" (UniqueName: \"kubernetes.io/projected/c367b1f5-858e-4c27-bcfc-56af079f456b-kube-api-access-v9497\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:59.549414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.549419 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c367b1f5-858e-4c27-bcfc-56af079f456b-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:59.549694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.549435 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkh42\" (UniqueName: \"kubernetes.io/projected/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c-kube-api-access-zkh42\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:33:59.878261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:33:59.878221 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.022671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.022632 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.152709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.152673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.254437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.253542 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.463471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.463375 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.500176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.485135 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" event={"ID":"f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c","Type":"ContainerDied","Data":"cf5ee026219243d6b0ad8b9f14afb982d0297dfa9d6971e07ed92b004176d60a"} Nov 03 20:34:00.500176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.485162 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th" Nov 03 20:34:00.500176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.485211 5043 scope.go:122] "RemoveContainer" containerID="c276c09521edb817935aea7eadd0e4345a4b4c8183c209328fd0b9d9944ec001" Nov 03 20:34:00.500176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.485685 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk" Nov 03 20:34:00.555013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.554884 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk"] Nov 03 20:34:00.564589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.564371 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fd01ad-5ccfbdf847-7b5kk"] Nov 03 20:34:00.594767 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.594722 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th"] Nov 03 20:34:00.598306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.598280 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fd01ad-7d99684dd-v77th"] Nov 03 20:34:00.682206 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.682170 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" path="/var/lib/kubelet/pods/c367b1f5-858e-4c27-bcfc-56af079f456b/volumes" Nov 03 20:34:00.682833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.682811 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" path="/var/lib/kubelet/pods/f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c/volumes" Nov 03 20:34:00.836133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.835118 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.936372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:00.934540 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.032737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:01.032693 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.156272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:01.156175 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.514714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:01.514624 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.704094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:01.703990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.415033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:02.414984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.862322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:02.862286 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:03.199691 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:03.199612 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:04.730914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:04.730878 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:05.457990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.457953 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:05.570464 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.570430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:05.630141 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:05.630104 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:34:05.768405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.768322 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8"] Nov 03 20:34:05.768780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.768547 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" containerID="cri-o://c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0" gracePeriod=30 Nov 03 20:34:05.781368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.781342 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg"] Nov 03 20:34:05.781529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.781510 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" containerID="cri-o://dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c" gracePeriod=30 Nov 03 20:34:05.799519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:05.799492 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.220472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.220427 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm"] Nov 03 20:34:06.221066 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.220692 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" podUID="afc6e0cd-9441-4d19-b553-19044c38ac0b" containerName="maas-api" containerID="cri-o://c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75" gracePeriod=30 Nov 03 20:34:06.280468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.280381 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.536585 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.536557 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:34:06.556207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.556160 5043 generic.go:356] "Generic (PLEG): container finished" podID="afc6e0cd-9441-4d19-b553-19044c38ac0b" containerID="c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75" exitCode=0 Nov 03 20:34:06.556736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.556299 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" event={"ID":"afc6e0cd-9441-4d19-b553-19044c38ac0b","Type":"ContainerDied","Data":"c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75"} Nov 03 20:34:06.556736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.556426 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" event={"ID":"afc6e0cd-9441-4d19-b553-19044c38ac0b","Type":"ContainerDied","Data":"a4c8b6ed844abe44812066991776f646000302690fe0a245b50cb16f725fac33"} Nov 03 20:34:06.556736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.556464 5043 scope.go:122] "RemoveContainer" containerID="c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75" Nov 03 20:34:06.556736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.556610 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm" Nov 03 20:34:06.601959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.601922 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft"] Nov 03 20:34:06.602835 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.602782 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" podUID="25078489-ee49-4746-b11b-b203c00e6787" containerName="maas-api" containerID="cri-o://21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c" gracePeriod=30 Nov 03 20:34:06.611110 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.606233 5043 scope.go:122] "RemoveContainer" containerID="c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75" Nov 03 20:34:06.649267 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:06.649233 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75\": container with ID starting with c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75 not found: ID does not exist" containerID="c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75" Nov 03 20:34:06.649482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.649453 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75"} err="failed to get container status \"c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75\": rpc error: code = NotFound desc = could not find container \"c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75\": container with ID starting with c3455aae97cfc4ee82e675a0b9b805b375ff4ae7cfa0e3ca49341b429e354e75 not found: ID does not exist" Nov 03 20:34:06.692101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.691216 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca\") pod \"afc6e0cd-9441-4d19-b553-19044c38ac0b\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " Nov 03 20:34:06.692101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.691303 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls\") pod \"afc6e0cd-9441-4d19-b553-19044c38ac0b\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " Nov 03 20:34:06.692101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.691587 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5\" (UniqueName: \"kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5\") pod \"afc6e0cd-9441-4d19-b553-19044c38ac0b\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " Nov 03 20:34:06.692101 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.691659 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs\") pod \"afc6e0cd-9441-4d19-b553-19044c38ac0b\" (UID: \"afc6e0cd-9441-4d19-b553-19044c38ac0b\") " Nov 03 20:34:06.713832 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.713766 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca" pod "afc6e0cd-9441-4d19-b553-19044c38ac0b" (UID: "afc6e0cd-9441-4d19-b553-19044c38ac0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:06.720514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.717478 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs" pod "afc6e0cd-9441-4d19-b553-19044c38ac0b" (UID: "afc6e0cd-9441-4d19-b553-19044c38ac0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:06.720514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.717916 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls" pod "afc6e0cd-9441-4d19-b553-19044c38ac0b" (UID: "afc6e0cd-9441-4d19-b553-19044c38ac0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:06.725096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.720818 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5" pod "afc6e0cd-9441-4d19-b553-19044c38ac0b" (UID: "afc6e0cd-9441-4d19-b553-19044c38ac0b"). InnerVolumeSpecName "kube-api-access-ff2d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:06.725096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.723106 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.848201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.837607 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2d5\" (UniqueName: \"kubernetes.io/projected/afc6e0cd-9441-4d19-b553-19044c38ac0b-kube-api-access-ff2d5\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:06.848201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.837645 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:06.848201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.837669 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6e0cd-9441-4d19-b553-19044c38ac0b-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:06.848201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.837684 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6e0cd-9441-4d19-b553-19044c38ac0b-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:06.928403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.928356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.964527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.964468 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm"] Nov 03 20:34:06.976214 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:06.974815 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9973d9-57d4fb6548-dfzqm"] Nov 03 20:34:07.028227 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.028171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:07.186447 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.186064 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:34:07.269742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.268556 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls\") pod \"25078489-ee49-4746-b11b-b203c00e6787\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " Nov 03 20:34:07.269742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.268650 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk\" (UniqueName: \"kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk\") pod \"25078489-ee49-4746-b11b-b203c00e6787\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " Nov 03 20:34:07.269742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.268856 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs\") pod \"25078489-ee49-4746-b11b-b203c00e6787\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " Nov 03 20:34:07.269742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.268892 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca\") pod \"25078489-ee49-4746-b11b-b203c00e6787\" (UID: \"25078489-ee49-4746-b11b-b203c00e6787\") " Nov 03 20:34:07.270597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.270275 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca" pod "25078489-ee49-4746-b11b-b203c00e6787" (UID: "25078489-ee49-4746-b11b-b203c00e6787"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:07.276502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.276218 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls" pod "25078489-ee49-4746-b11b-b203c00e6787" (UID: "25078489-ee49-4746-b11b-b203c00e6787"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:07.286273 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.286234 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs" pod "25078489-ee49-4746-b11b-b203c00e6787" (UID: "25078489-ee49-4746-b11b-b203c00e6787"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:07.295471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.295444 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk" pod "25078489-ee49-4746-b11b-b203c00e6787" (UID: "25078489-ee49-4746-b11b-b203c00e6787"). InnerVolumeSpecName "kube-api-access-574xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:07.370818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.370778 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:07.370818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.370814 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574xk\" (UniqueName: \"kubernetes.io/projected/25078489-ee49-4746-b11b-b203c00e6787-kube-api-access-574xk\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:07.371018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.370827 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25078489-ee49-4746-b11b-b203c00e6787-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:07.371018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.370840 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25078489-ee49-4746-b11b-b203c00e6787-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:07.474774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.474685 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:07.565109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.565036 5043 generic.go:356] "Generic (PLEG): container finished" podID="25078489-ee49-4746-b11b-b203c00e6787" containerID="21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c" exitCode=0 Nov 03 20:34:07.565109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.565101 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" Nov 03 20:34:07.565324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.565130 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" event={"ID":"25078489-ee49-4746-b11b-b203c00e6787","Type":"ContainerDied","Data":"21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c"} Nov 03 20:34:07.565324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.565187 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft" event={"ID":"25078489-ee49-4746-b11b-b203c00e6787","Type":"ContainerDied","Data":"37bbea8078a524724f859fd993129ea43c5260398991ca69e372fbdfac669ef3"} Nov 03 20:34:07.565324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.565218 5043 scope.go:122] "RemoveContainer" containerID="21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c" Nov 03 20:34:07.576846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.576828 5043 scope.go:122] "RemoveContainer" containerID="21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c" Nov 03 20:34:07.577196 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:07.577176 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c\": container with ID starting with 21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c not found: ID does not exist" containerID="21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c" Nov 03 20:34:07.577265 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.577206 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c"} err="failed to get container status \"21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c\": rpc error: code = NotFound desc = could not find container \"21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c\": container with ID starting with 21ad90e3ee0cfab701a11d209426db939fd85a9a1ed43b374a1599db7b0ef53c not found: ID does not exist" Nov 03 20:34:07.638197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.638164 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft"] Nov 03 20:34:07.651601 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.651570 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fd01ad-7b5cf9c9f6-mg6ft"] Nov 03 20:34:07.691383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.691272 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:07.796740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:07.796651 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:08.102965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:08.102929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:08.677959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:08.677922 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25078489-ee49-4746-b11b-b203c00e6787" path="/var/lib/kubelet/pods/25078489-ee49-4746-b11b-b203c00e6787/volumes" Nov 03 20:34:08.678677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:08.678649 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc6e0cd-9441-4d19-b553-19044c38ac0b" path="/var/lib/kubelet/pods/afc6e0cd-9441-4d19-b553-19044c38ac0b/volumes" Nov 03 20:34:09.131008 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.130975 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:09.209099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.203874 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" containerID="cri-o://acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e" gracePeriod=45 Nov 03 20:34:09.232895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.232866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:09.453491 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.453451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:09.503336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.503212 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:34:09.589207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.587715 5043 generic.go:356] "Generic (PLEG): container finished" podID="3144a231-859a-4528-9c96-d9e60053adc8" containerID="acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e" exitCode=0 Nov 03 20:34:09.589207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.587956 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" event={"ID":"3144a231-859a-4528-9c96-d9e60053adc8","Type":"ContainerDied","Data":"acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e"} Nov 03 20:34:09.589207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.587991 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" event={"ID":"3144a231-859a-4528-9c96-d9e60053adc8","Type":"ContainerDied","Data":"f34984cde3a23a6d1235cf2f6d1e3fb11658cc8479482f1d11bd75b13182ced6"} Nov 03 20:34:09.589207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.588013 5043 scope.go:122] "RemoveContainer" containerID="acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e" Nov 03 20:34:09.589207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.588145 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s" Nov 03 20:34:09.605746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.605724 5043 scope.go:122] "RemoveContainer" containerID="acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e" Nov 03 20:34:09.606095 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:09.606051 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e\": container with ID starting with acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e not found: ID does not exist" containerID="acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e" Nov 03 20:34:09.606182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.606109 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e"} err="failed to get container status \"acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e\": rpc error: code = NotFound desc = could not find container \"acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e\": container with ID starting with acbb75fc5c2578bfd23532dc0148b9c3e3a3c57506c9c0f25d9c6d6c67316f1e not found: ID does not exist" Nov 03 20:34:09.615467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.615439 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs\") pod \"3144a231-859a-4528-9c96-d9e60053adc8\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " Nov 03 20:34:09.615586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.615515 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm\") pod \"3144a231-859a-4528-9c96-d9e60053adc8\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " Nov 03 20:34:09.615665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.615603 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir\") pod \"3144a231-859a-4528-9c96-d9e60053adc8\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " Nov 03 20:34:09.615665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.615656 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf\" (UniqueName: \"kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf\") pod \"3144a231-859a-4528-9c96-d9e60053adc8\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " Nov 03 20:34:09.615788 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.615722 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache\") pod \"3144a231-859a-4528-9c96-d9e60053adc8\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " Nov 03 20:34:09.615906 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.615879 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir" pod "3144a231-859a-4528-9c96-d9e60053adc8" (UID: "3144a231-859a-4528-9c96-d9e60053adc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:09.616155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.616131 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache" pod "3144a231-859a-4528-9c96-d9e60053adc8" (UID: "3144a231-859a-4528-9c96-d9e60053adc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:09.616322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.616284 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home" pod "3144a231-859a-4528-9c96-d9e60053adc8" (UID: "3144a231-859a-4528-9c96-d9e60053adc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:09.618252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.618231 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm" pod "3144a231-859a-4528-9c96-d9e60053adc8" (UID: "3144a231-859a-4528-9c96-d9e60053adc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:09.618414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.618397 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf" pod "3144a231-859a-4528-9c96-d9e60053adc8" (UID: "3144a231-859a-4528-9c96-d9e60053adc8"). InnerVolumeSpecName "kube-api-access-hfkzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:09.619149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.619125 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home\") pod \"3144a231-859a-4528-9c96-d9e60053adc8\" (UID: \"3144a231-859a-4528-9c96-d9e60053adc8\") " Nov 03 20:34:09.619750 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.619732 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-tmp-dir\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:09.619840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.619757 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfkzf\" (UniqueName: \"kubernetes.io/projected/3144a231-859a-4528-9c96-d9e60053adc8-kube-api-access-hfkzf\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:09.619840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.619771 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-model-cache\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:09.619840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.619783 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-home\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:09.619840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.619797 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3144a231-859a-4528-9c96-d9e60053adc8-dshm\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:09.621201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.621173 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs" pod "3144a231-859a-4528-9c96-d9e60053adc8" (UID: "3144a231-859a-4528-9c96-d9e60053adc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:09.721389 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.721300 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3144a231-859a-4528-9c96-d9e60053adc8-tls-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:09.936801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.936769 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s"] Nov 03 20:34:09.949434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:09.949406 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-w0-e40816-kserve-6b4c76b47-9qv4s"] Nov 03 20:34:10.678438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:10.678396 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.679295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:10.679271 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3144a231-859a-4528-9c96-d9e60053adc8" path="/var/lib/kubelet/pods/3144a231-859a-4528-9c96-d9e60053adc8/volumes" Nov 03 20:34:10.797379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:10.797346 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:11.093665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.093639 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:34:11.146281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.146226 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:11.170501 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.170470 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:34:11.248392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248295 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.248546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248474 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.248601 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248586 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.248640 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248629 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.248684 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248672 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.248726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248711 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248763 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248793 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248856 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.248899 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token\") pod \"076e74c9-625f-40d8-a4a3-0488b0f6c228\" (UID: \"076e74c9-625f-40d8-a4a3-0488b0f6c228\") " Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249015 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249036 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249248 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249466 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249486 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249501 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076e74c9-625f-40d8-a4a3-0488b0f6c228-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249509 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.249578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.249554 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.251426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.251399 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:11.251534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.251449 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.251534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.251487 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "kube-api-access-htrvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:11.251534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.251514 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token" pod "076e74c9-625f-40d8-a4a3-0488b0f6c228" (UID: "076e74c9-625f-40d8-a4a3-0488b0f6c228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:11.350575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350541 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.350755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350597 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.350755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350631 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.350755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350675 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.350755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350713 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.350755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350755 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.351057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350781 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.351057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350859 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.351057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.350894 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs\") pod \"6503bafd-9083-451e-b3a2-d8245349fbe1\" (UID: \"6503bafd-9083-451e-b3a2-d8245349fbe1\") " Nov 03 20:34:11.351268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351155 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.351559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351453 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.351559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351529 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351157 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351635 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351640 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351649 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351688 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrvv\" (UniqueName: \"kubernetes.io/projected/076e74c9-625f-40d8-a4a3-0488b0f6c228-kube-api-access-htrvv\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351708 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.351725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351727 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.352126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.351743 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076e74c9-625f-40d8-a4a3-0488b0f6c228-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.352874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.352851 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "kube-api-access-ldnx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:11.352980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.352910 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:11.353321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.353294 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:11.353756 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.353740 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token" pod "6503bafd-9083-451e-b3a2-d8245349fbe1" (UID: "6503bafd-9083-451e-b3a2-d8245349fbe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:11.453057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453023 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453050 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453060 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453057 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453085 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6503bafd-9083-451e-b3a2-d8245349fbe1-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453319 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453096 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldnx6\" (UniqueName: \"kubernetes.io/projected/6503bafd-9083-451e-b3a2-d8245349fbe1-kube-api-access-ldnx6\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453319 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453107 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453319 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453116 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6503bafd-9083-451e-b3a2-d8245349fbe1-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.453319 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.453124 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6503bafd-9083-451e-b3a2-d8245349fbe1-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:11.605145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.605103 5043 generic.go:356] "Generic (PLEG): container finished" podID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerID="c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0" exitCode=0 Nov 03 20:34:11.605338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.605161 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" Nov 03 20:34:11.605338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.605208 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" event={"ID":"076e74c9-625f-40d8-a4a3-0488b0f6c228","Type":"ContainerDied","Data":"c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0"} Nov 03 20:34:11.605338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.605282 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8" event={"ID":"076e74c9-625f-40d8-a4a3-0488b0f6c228","Type":"ContainerDied","Data":"60fddda94bb3e74140410404ad8cdab8a24a647a21d017a155e1cd7c928ab262"} Nov 03 20:34:11.605338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.605310 5043 scope.go:122] "RemoveContainer" containerID="c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0" Nov 03 20:34:11.607905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.607762 5043 generic.go:356] "Generic (PLEG): container finished" podID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerID="dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c" exitCode=0 Nov 03 20:34:11.607905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.607827 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" event={"ID":"6503bafd-9083-451e-b3a2-d8245349fbe1","Type":"ContainerDied","Data":"dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c"} Nov 03 20:34:11.607905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.607837 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" Nov 03 20:34:11.607905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.607864 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg" event={"ID":"6503bafd-9083-451e-b3a2-d8245349fbe1","Type":"ContainerDied","Data":"e03047881b551f053822630c13ab84372966686b8ba40d47bff12dfde3ff30ed"} Nov 03 20:34:11.619643 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.619625 5043 scope.go:122] "RemoveContainer" containerID="c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0" Nov 03 20:34:11.619957 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:11.619935 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0\": container with ID starting with c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0 not found: ID does not exist" containerID="c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0" Nov 03 20:34:11.620064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.619970 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0"} err="failed to get container status \"c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0\": rpc error: code = NotFound desc = could not find container \"c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0\": container with ID starting with c5f447ef240992e483edf2201ddc868200a31ada445c14b8b244264c05bb83e0 not found: ID does not exist" Nov 03 20:34:11.620064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.619993 5043 scope.go:122] "RemoveContainer" containerID="dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c" Nov 03 20:34:11.630402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.630369 5043 scope.go:122] "RemoveContainer" containerID="dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c" Nov 03 20:34:11.630671 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:11.630651 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c\": container with ID starting with dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c not found: ID does not exist" containerID="dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c" Nov 03 20:34:11.630742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.630680 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c"} err="failed to get container status \"dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c\": rpc error: code = NotFound desc = could not find container \"dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c\": container with ID starting with dbf2e2bf2045163ff0a00df4ecd7a197987f04a011e9402768ed9e7693954f6c not found: ID does not exist" Nov 03 20:34:11.660489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.660469 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg"] Nov 03 20:34:11.666528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.666509 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sk2vg"] Nov 03 20:34:11.704735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.704699 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8"] Nov 03 20:34:11.711804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:11.711779 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-2rpg8"] Nov 03 20:34:12.607874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:12.607835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:12.682225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:12.680477 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" path="/var/lib/kubelet/pods/076e74c9-625f-40d8-a4a3-0488b0f6c228/volumes" Nov 03 20:34:12.682225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:12.681177 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" path="/var/lib/kubelet/pods/6503bafd-9083-451e-b3a2-d8245349fbe1/volumes" Nov 03 20:34:13.959513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:13.959479 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.054645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.054614 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.269919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.269830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.288969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.288934 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t"] Nov 03 20:34:14.289266 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.289232 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" containerID="cri-o://c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984" gracePeriod=30 Nov 03 20:34:14.341894 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.341854 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd"] Nov 03 20:34:14.342130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.342056 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" containerID="cri-o://5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc" gracePeriod=30 Nov 03 20:34:14.371856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.371830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.585561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.585523 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.685751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:14.685711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:15.016087 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:15.015864 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn"] Nov 03 20:34:15.016651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:15.016150 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" containerID="cri-o://5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb" gracePeriod=30 Nov 03 20:34:15.044025 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:15.043997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:16.008422 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:16.008388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.487553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.487521 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz"] Nov 03 20:34:17.488159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488142 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ddad4d1d-a61a-457e-8523-b4021250089f" containerName="payload-pre-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488158 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddad4d1d-a61a-457e-8523-b4021250089f" containerName="payload-pre-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488169 5043 cpu_manager.go:414] "containerMap: removing container" podUID="092a749e-6b8c-47b0-a314-5f1347057618" containerName="payload-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488175 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092a749e-6b8c-47b0-a314-5f1347057618" containerName="payload-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488184 5043 cpu_manager.go:414] "containerMap: removing container" podUID="8f4eced3-9286-4b3a-9d75-0fe80735f8b5" containerName="maas-api" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488188 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4eced3-9286-4b3a-9d75-0fe80735f8b5" containerName="maas-api" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488195 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" containerName="payload-pre-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488200 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" containerName="payload-pre-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488207 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c6eae762-603f-471d-8b1a-67fdb9fb6db8" containerName="payload-pre-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488212 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eae762-603f-471d-8b1a-67fdb9fb6db8" containerName="payload-pre-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488223 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6b72babb-158e-496a-bed3-3d6dfa3e1a6e" containerName="payload-processing" Nov 03 20:34:17.488223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488228 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b72babb-158e-496a-bed3-3d6dfa3e1a6e" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488236 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488240 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488246 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1a7473c1-6c0e-4644-89dd-3124415127b2" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488251 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7473c1-6c0e-4644-89dd-3124415127b2" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488260 5043 cpu_manager.go:414] "containerMap: removing container" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488264 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488273 5043 cpu_manager.go:414] "containerMap: removing container" podUID="05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488278 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488288 5043 cpu_manager.go:414] "containerMap: removing container" podUID="afc6e0cd-9441-4d19-b553-19044c38ac0b" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488292 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc6e0cd-9441-4d19-b553-19044c38ac0b" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488298 5043 cpu_manager.go:414] "containerMap: removing container" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488302 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488324 5043 cpu_manager.go:414] "containerMap: removing container" podUID="bafe64f1-b7d6-4819-ac96-1d6851b30dde" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488329 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafe64f1-b7d6-4819-ac96-1d6851b30dde" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488338 5043 cpu_manager.go:414] "containerMap: removing container" podUID="098e1578-d007-4299-82cf-d9e05369f350" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488343 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098e1578-d007-4299-82cf-d9e05369f350" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488350 5043 cpu_manager.go:414] "containerMap: removing container" podUID="25078489-ee49-4746-b11b-b203c00e6787" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488354 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25078489-ee49-4746-b11b-b203c00e6787" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488361 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a64147b6-3d77-485d-979f-09b3cdbdfe49" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488367 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64147b6-3d77-485d-979f-09b3cdbdfe49" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488375 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488379 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488386 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488390 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488396 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488400 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488406 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488411 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488418 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488422 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488434 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488438 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488445 5043 cpu_manager.go:414] "containerMap: removing container" podUID="53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488449 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488455 5043 cpu_manager.go:414] "containerMap: removing container" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488459 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488465 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2045fa93-9dec-4d08-9d76-2dc4122e1306" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488469 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2045fa93-9dec-4d08-9d76-2dc4122e1306" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488476 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488481 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488487 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488492 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488500 5043 cpu_manager.go:414] "containerMap: removing container" podUID="277c75ae-0568-4c6c-bcfb-b574312c43ca" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488503 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277c75ae-0568-4c6c-bcfb-b574312c43ca" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488518 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488522 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488532 5043 cpu_manager.go:414] "containerMap: removing container" podUID="10969fd1-504d-4fc7-8236-73fde6043259" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488536 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10969fd1-504d-4fc7-8236-73fde6043259" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488543 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2d9be17b-23c3-446b-ae0b-ca419c143d1d" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488548 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9be17b-23c3-446b-ae0b-ca419c143d1d" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488637 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c367b1f5-858e-4c27-bcfc-56af079f456b" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488648 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="10969fd1-504d-4fc7-8236-73fde6043259" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488655 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9be17b-23c3-446b-ae0b-ca419c143d1d" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488660 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d145721-a33b-45d5-8107-399a9f25f4b5" containerName="cleanup" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488666 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddad4d1d-a61a-457e-8523-b4021250089f" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488672 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="092a749e-6b8c-47b0-a314-5f1347057618" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488677 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cbeb0f-924d-4c06-bb3d-bdd0c00999cb" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488684 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4eced3-9286-4b3a-9d75-0fe80735f8b5" containerName="maas-api" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488689 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eae762-603f-471d-8b1a-67fdb9fb6db8" containerName="payload-pre-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488695 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b72babb-158e-496a-bed3-3d6dfa3e1a6e" containerName="payload-processing" Nov 03 20:34:17.488663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488702 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6503bafd-9083-451e-b3a2-d8245349fbe1" containerName="istio-proxy" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488708 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7473c1-6c0e-4644-89dd-3124415127b2" containerName="payload-processing" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488713 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="334cc1ee-0aa4-451f-b1e8-0be38daaff49" containerName="main" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488720 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ed4fae-bc77-4a91-ab7e-5d5dfe4fbc02" containerName="maas-api" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488726 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc6e0cd-9441-4d19-b553-19044c38ac0b" containerName="maas-api" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488731 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b66086-db00-4280-8a94-e515abf810ab" containerName="istio-proxy" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488738 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafe64f1-b7d6-4819-ac96-1d6851b30dde" containerName="payload-pre-processing" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488744 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="098e1578-d007-4299-82cf-d9e05369f350" containerName="payload-pre-processing" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488750 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="25078489-ee49-4746-b11b-b203c00e6787" containerName="maas-api" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488755 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64147b6-3d77-485d-979f-09b3cdbdfe49" containerName="payload-processing" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488761 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3144a231-859a-4528-9c96-d9e60053adc8" containerName="main" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488766 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf09f94-edfa-4003-ab49-2e3e3e8cf8c8" containerName="maas-api" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488771 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39f3aa8-1aa5-4a26-a91c-e9b93187bffd" containerName="maas-api" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488778 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eff5c5d-7869-41ea-b7fe-f499411cec5a" containerName="istio-proxy" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488784 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f935dcc5-d66f-4c4e-8a2f-f844f2c3dd4c" containerName="payload-processing" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488789 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="076e74c9-625f-40d8-a4a3-0488b0f6c228" containerName="istio-proxy" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488793 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="53011f9e-2a42-4c14-a8e5-f8c45d12bf3c" containerName="payload-processing" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488799 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2045fa93-9dec-4d08-9d76-2dc4122e1306" containerName="maas-api" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488804 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd7e46d-55f9-4d5e-b19e-b1f93c3f6f97" containerName="istio-proxy" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488809 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ae8950-3221-420c-b2eb-fd4035002a6f" containerName="istio-proxy" Nov 03 20:34:17.490591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.488814 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="277c75ae-0568-4c6c-bcfb-b574312c43ca" containerName="maas-api" Nov 03 20:34:17.494478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.494458 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.509575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.509143 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4b95f3-serving-cert\"" Nov 03 20:34:17.527826 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.527794 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz"] Nov 03 20:34:17.537037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.536999 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nffx\" (UniqueName: \"kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.537262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.537172 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.537262 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.537208 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.537420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.537306 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.569534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.569505 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.638411 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.638383 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nffx\" (UniqueName: \"kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.638561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.638474 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.638561 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.638519 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.638682 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.638664 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.639295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.639276 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.640998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.640978 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.641105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.641016 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.659269 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.659245 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nffx\" (UniqueName: \"kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx\") pod \"maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:17.669678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.669648 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.808698 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:17.808618 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:18.116606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:18.116574 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz"] Nov 03 20:34:18.120392 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:34:18.120356 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda502b248_8079_481a_8837_f598a2f1829a.slice/crio-354ea3708d86162e56af40f6327107022cb804b5689bdfad257978c3d6ab15c6 WatchSource:0}: Error finding container 354ea3708d86162e56af40f6327107022cb804b5689bdfad257978c3d6ab15c6: Status 404 returned error can't find the container with id 354ea3708d86162e56af40f6327107022cb804b5689bdfad257978c3d6ab15c6 Nov 03 20:34:18.122234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:18.122212 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:34:18.664271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:18.664233 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" event={"ID":"a502b248-8079-481a-8837-f598a2f1829a","Type":"ContainerStarted","Data":"87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab"} Nov 03 20:34:18.665282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:18.664293 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" event={"ID":"a502b248-8079-481a-8837-f598a2f1829a","Type":"ContainerStarted","Data":"354ea3708d86162e56af40f6327107022cb804b5689bdfad257978c3d6ab15c6"} Nov 03 20:34:18.665282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:18.664324 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:18.705717 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:18.705660 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" podStartSLOduration=1.310728644 podStartE2EDuration="1.705641748s" totalImagesPullingTime="394.913104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:18.703177671 +0000 UTC m=+1840.170132938" watchObservedRunningTime="2026-11-03 20:34:18.705641748 +0000 UTC m=+1840.172597000" Nov 03 20:34:19.164700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.164668 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:34:19.164900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.164724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:34:19.313592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.313555 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.562591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.562562 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:34:19.636607 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.636582 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.669863 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670021 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670062 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670143 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670193 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670274 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670309 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670359 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.672920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.670428 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data\") pod \"e2859563-48c2-429d-ba23-b22f5b16b30e\" (UID: \"e2859563-48c2-429d-ba23-b22f5b16b30e\") " Nov 03 20:34:19.674728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.673960 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.674728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.674201 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.677201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.677163 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:19.677795 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.677772 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.678536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.678187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.678536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.678497 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.680804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.679425 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:19.680804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.680762 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "kube-api-access-bd4vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:19.681003 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.680972 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token" pod "e2859563-48c2-429d-ba23-b22f5b16b30e" (UID: "e2859563-48c2-429d-ba23-b22f5b16b30e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:19.759218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.736086 5043 generic.go:356] "Generic (PLEG): container finished" podID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerID="5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc" exitCode=0 Nov 03 20:34:19.759218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.736341 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" event={"ID":"eb39e4e7-e05b-4abd-b40f-24a79aeb7696","Type":"ContainerDied","Data":"5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc"} Nov 03 20:34:19.759218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.736398 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" event={"ID":"eb39e4e7-e05b-4abd-b40f-24a79aeb7696","Type":"ContainerDied","Data":"d3adfe6e1013a0cac5ab98029f32f5037770c41c3babb7352289cd0cd4c2b599"} Nov 03 20:34:19.759218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.736428 5043 scope.go:122] "RemoveContainer" containerID="5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc" Nov 03 20:34:19.759218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.736587 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd" Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.776435 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.776582 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.776816 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.776854 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.776949 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.777165 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.777236 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.777418 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.777568 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token\") pod \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\" (UID: \"eb39e4e7-e05b-4abd-b40f-24a79aeb7696\") " Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.778271 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.778491 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.778689 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.782434 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.785004 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.784252 5043 generic.go:356] "Generic (PLEG): container finished" podID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerID="c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984" exitCode=0 Nov 03 20:34:19.791150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.787429 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" Nov 03 20:34:19.791150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.787776 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" event={"ID":"e2859563-48c2-429d-ba23-b22f5b16b30e","Type":"ContainerDied","Data":"c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984"} Nov 03 20:34:19.791150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.787873 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t" event={"ID":"e2859563-48c2-429d-ba23-b22f5b16b30e","Type":"ContainerDied","Data":"e5ebb19daed9c9310874e487744603d7f0e66a830e3dc973ff0c0e1e519282ce"} Nov 03 20:34:19.791150 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.788529 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:19.791469 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.791278 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.809595 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.809741 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.810150 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d" pod "eb39e4e7-e05b-4abd-b40f-24a79aeb7696" (UID: "eb39e4e7-e05b-4abd-b40f-24a79aeb7696"). InnerVolumeSpecName "kube-api-access-b9d4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.812645 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.812934 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2859563-48c2-429d-ba23-b22f5b16b30e-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.812953 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4vn\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-kube-api-access-bd4vn\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.812973 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.812988 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813001 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813015 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813033 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813045 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2859563-48c2-429d-ba23-b22f5b16b30e-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813058 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2859563-48c2-429d-ba23-b22f5b16b30e-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813159 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.815337 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.813181 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.822623 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.822601 5043 scope.go:122] "RemoveContainer" containerID="5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc" Nov 03 20:34:19.822958 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:19.822937 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc\": container with ID starting with 5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc not found: ID does not exist" containerID="5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc" Nov 03 20:34:19.823036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.822966 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc"} err="failed to get container status \"5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc\": rpc error: code = NotFound desc = could not find container \"5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc\": container with ID starting with 5041caa3d3a4a5d2472a333d4575f53f2e8fc84f1b1a3b728d047df5082ffdcc not found: ID does not exist" Nov 03 20:34:19.823036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.822983 5043 scope.go:122] "RemoveContainer" containerID="c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984" Nov 03 20:34:19.833109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.833088 5043 scope.go:122] "RemoveContainer" containerID="c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984" Nov 03 20:34:19.833392 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:19.833373 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984\": container with ID starting with c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984 not found: ID does not exist" containerID="c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984" Nov 03 20:34:19.833457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.833399 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984"} err="failed to get container status \"c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984\": rpc error: code = NotFound desc = could not find container \"c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984\": container with ID starting with c36bc27bf3cacc6f39ff3bc9bce04255094162e53c247acc4883a113186b8984 not found: ID does not exist" Nov 03 20:34:19.860212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.860191 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t"] Nov 03 20:34:19.863485 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.863462 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-btp6t"] Nov 03 20:34:19.914383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.914362 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9d4d\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-kube-api-access-b9d4d\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.914383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.914383 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.914553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.914394 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.914553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.914402 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.914553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.914410 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:19.914553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:19.914419 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb39e4e7-e05b-4abd-b40f-24a79aeb7696-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.135514 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.135311 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd"] Nov 03 20:34:20.140783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.140759 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-mlftd"] Nov 03 20:34:20.279166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.279140 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:34:20.420454 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420422 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420489 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420513 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420549 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420651 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420635 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420674 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420704 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420752 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.420879 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.420832 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket\") pod \"a3ffd880-d006-42cf-b4e5-93884993ca51\" (UID: \"a3ffd880-d006-42cf-b4e5-93884993ca51\") " Nov 03 20:34:20.421631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.421045 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:20.421631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.421508 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ffd880-d006-42cf-b4e5-93884993ca51-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.421824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.421696 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:20.421824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.421710 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:20.421824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.421753 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:20.421824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.421784 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:20.423281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.423259 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "kube-api-access-7gm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:20.423869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.423845 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:20.424193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.424173 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:20.424321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.424298 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo" pod "a3ffd880-d006-42cf-b4e5-93884993ca51" (UID: "a3ffd880-d006-42cf-b4e5-93884993ca51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:20.523020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.522979 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gm2q\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-kube-api-access-7gm2q\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523011 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523022 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523030 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523040 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523051 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523064 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.523348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.523116 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ffd880-d006-42cf-b4e5-93884993ca51-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:20.683291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.683227 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" path="/var/lib/kubelet/pods/e2859563-48c2-429d-ba23-b22f5b16b30e/volumes" Nov 03 20:34:20.683715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.683700 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" path="/var/lib/kubelet/pods/eb39e4e7-e05b-4abd-b40f-24a79aeb7696/volumes" Nov 03 20:34:20.795481 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.795445 5043 generic.go:356] "Generic (PLEG): container finished" podID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerID="5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb" exitCode=0 Nov 03 20:34:20.795677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.795493 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" Nov 03 20:34:20.795677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.795542 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" event={"ID":"a3ffd880-d006-42cf-b4e5-93884993ca51","Type":"ContainerDied","Data":"5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb"} Nov 03 20:34:20.795677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.795600 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn" event={"ID":"a3ffd880-d006-42cf-b4e5-93884993ca51","Type":"ContainerDied","Data":"30ed626f942604e10ac4be4453d6fb715fb45fd1073a201edd028a93d269ea9d"} Nov 03 20:34:20.795677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.795629 5043 scope.go:122] "RemoveContainer" containerID="5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb" Nov 03 20:34:20.806109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.806090 5043 scope.go:122] "RemoveContainer" containerID="5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb" Nov 03 20:34:20.806407 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:20.806384 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb\": container with ID starting with 5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb not found: ID does not exist" containerID="5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb" Nov 03 20:34:20.806480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.806418 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb"} err="failed to get container status \"5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb\": rpc error: code = NotFound desc = could not find container \"5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb\": container with ID starting with 5f695dcfaf7b40e65ce3e81e93b44716e0d3bb09ef7e9479654e005cef9758fb not found: ID does not exist" Nov 03 20:34:20.851280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.851253 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn"] Nov 03 20:34:20.851409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:20.851293 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-jb5sn"] Nov 03 20:34:21.002427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:21.002327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.167551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:22.167517 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.273999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:22.273934 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.520203 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:22.520129 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.678316 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:22.678280 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" path="/var/lib/kubelet/pods/a3ffd880-d006-42cf-b4e5-93884993ca51/volumes" Nov 03 20:34:24.021971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:24.021939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.795548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:24.795519 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:34:25.236506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.236467 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t"] Nov 03 20:34:25.236986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.236738 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" podUID="51dfcaed-031e-4192-b2fb-e3a7a7cab90d" containerName="maas-api" containerID="cri-o://031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c" gracePeriod=30 Nov 03 20:34:25.303141 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.301741 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:25.572324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.572288 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:34:25.608627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.608588 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls\") pod \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " Nov 03 20:34:25.608627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.608657 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp\" (UniqueName: \"kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp\") pod \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " Nov 03 20:34:25.608911 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.608772 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs\") pod \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " Nov 03 20:34:25.608911 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.608860 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca\") pod \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\" (UID: \"51dfcaed-031e-4192-b2fb-e3a7a7cab90d\") " Nov 03 20:34:25.609395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.609368 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca" pod "51dfcaed-031e-4192-b2fb-e3a7a7cab90d" (UID: "51dfcaed-031e-4192-b2fb-e3a7a7cab90d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:25.609597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.609578 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:25.625173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.617606 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls" pod "51dfcaed-031e-4192-b2fb-e3a7a7cab90d" (UID: "51dfcaed-031e-4192-b2fb-e3a7a7cab90d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:25.625173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.617716 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp" pod "51dfcaed-031e-4192-b2fb-e3a7a7cab90d" (UID: "51dfcaed-031e-4192-b2fb-e3a7a7cab90d"). InnerVolumeSpecName "kube-api-access-ttggp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:25.716780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.715759 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:25.716780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.715989 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttggp\" (UniqueName: \"kubernetes.io/projected/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-kube-api-access-ttggp\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:25.719843 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.719803 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs" pod "51dfcaed-031e-4192-b2fb-e3a7a7cab90d" (UID: "51dfcaed-031e-4192-b2fb-e3a7a7cab90d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:25.816932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.816854 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51dfcaed-031e-4192-b2fb-e3a7a7cab90d-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:25.832207 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.832171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:25.836888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.836858 5043 generic.go:356] "Generic (PLEG): container finished" podID="51dfcaed-031e-4192-b2fb-e3a7a7cab90d" containerID="031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c" exitCode=0 Nov 03 20:34:25.837011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.836906 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" Nov 03 20:34:25.837011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.836936 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" event={"ID":"51dfcaed-031e-4192-b2fb-e3a7a7cab90d","Type":"ContainerDied","Data":"031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c"} Nov 03 20:34:25.837011 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.836992 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t" event={"ID":"51dfcaed-031e-4192-b2fb-e3a7a7cab90d","Type":"ContainerDied","Data":"285c66b1c04810543bec534bb7c2ceb6ba0dad9db4e2362bc38793ec165cba82"} Nov 03 20:34:25.837138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.837017 5043 scope.go:122] "RemoveContainer" containerID="031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c" Nov 03 20:34:25.849510 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.849491 5043 scope.go:122] "RemoveContainer" containerID="031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c" Nov 03 20:34:25.849824 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:25.849796 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c\": container with ID starting with 031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c not found: ID does not exist" containerID="031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c" Nov 03 20:34:25.849917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.849834 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c"} err="failed to get container status \"031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c\": rpc error: code = NotFound desc = could not find container \"031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c\": container with ID starting with 031aff32488c232868cab6a6926e50678b04d1d83a7e44989a0d8c494867182c not found: ID does not exist" Nov 03 20:34:25.891721 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.891692 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t"] Nov 03 20:34:25.903687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.903663 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e40816-5bfc7d6f95-pl29t"] Nov 03 20:34:25.931766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:25.931737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:26.032625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:26.032592 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:26.407676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:26.407633 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:26.678149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:26.678118 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51dfcaed-031e-4192-b2fb-e3a7a7cab90d" path="/var/lib/kubelet/pods/51dfcaed-031e-4192-b2fb-e3a7a7cab90d/volumes" Nov 03 20:34:27.634161 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:27.634121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:29.392906 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:29.392639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.023823 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:31.023786 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.249934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:31.249861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.351987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:31.351953 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:32.074589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:32.074554 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:32.204050 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:32.204020 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:32.736480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:32.736442 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:34.175020 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:34.174983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:34.363264 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:34.363218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:34.484097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:34.483362 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:34.696672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:34.696632 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq"] Nov 03 20:34:34.696869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:34.696825 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" containerID="cri-o://7aa33a2dcf23cf73fb2bb2667c85d2c98a4f96d32b0b90d29546da6f3d5c0e47" gracePeriod=30 Nov 03 20:34:34.710890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:34.710842 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:35.630085 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:34:35.630036 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:34:36.092787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:36.092750 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.701863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:37.701826 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:38.710102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:38.710039 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:39.301507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:39.301467 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:39.542154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:39.542123 5043 scope.go:122] "RemoveContainer" containerID="b31767f6f593d6cf34b2e64cff7b7ff60df272dc5360a9748aed242e330bb511" Nov 03 20:34:39.961657 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:39.961618 5043 generic.go:356] "Generic (PLEG): container finished" podID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerID="7aa33a2dcf23cf73fb2bb2667c85d2c98a4f96d32b0b90d29546da6f3d5c0e47" exitCode=0 Nov 03 20:34:39.962115 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:39.961686 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" event={"ID":"753bcdc7-60a4-4d92-9ca4-e11a33d4e339","Type":"ContainerDied","Data":"7aa33a2dcf23cf73fb2bb2667c85d2c98a4f96d32b0b90d29546da6f3d5c0e47"} Nov 03 20:34:40.002543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.002519 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:34:40.115277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115237 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115344 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115407 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115451 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115543 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115501 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:40.115650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115551 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115584 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115761 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115669 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115761 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115738 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115872 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115812 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo\") pod \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\" (UID: \"753bcdc7-60a4-4d92-9ca4-e11a33d4e339\") " Nov 03 20:34:40.115928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115799 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:40.115977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.115905 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:40.116243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.116220 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:40.116460 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.116437 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:40.116597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.116488 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.116671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.116610 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.116671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.116624 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.116671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.116641 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.118356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.118325 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "kube-api-access-nsrcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:40.118430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.118362 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:40.118430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.118378 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:40.118676 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.118654 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo" pod "753bcdc7-60a4-4d92-9ca4-e11a33d4e339" (UID: "753bcdc7-60a4-4d92-9ca4-e11a33d4e339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:40.221486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.221450 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.221486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.221481 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.221486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.221494 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.221797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.221506 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsrcp\" (UniqueName: \"kubernetes.io/projected/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-kube-api-access-nsrcp\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.221797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.221522 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753bcdc7-60a4-4d92-9ca4-e11a33d4e339-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:34:40.971537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.971457 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" Nov 03 20:34:40.971537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.971473 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq" event={"ID":"753bcdc7-60a4-4d92-9ca4-e11a33d4e339","Type":"ContainerDied","Data":"bb4122ed9ddb178891bd2ac860b7e4c2d1774c2c45b8c7686ed4033eed5bca11"} Nov 03 20:34:40.972024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:40.971546 5043 scope.go:122] "RemoveContainer" containerID="7aa33a2dcf23cf73fb2bb2667c85d2c98a4f96d32b0b90d29546da6f3d5c0e47" Nov 03 20:34:41.036142 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:41.036113 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq"] Nov 03 20:34:41.047113 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:41.047085 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5vrtq"] Nov 03 20:34:41.095427 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:41.095397 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.213548 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:41.213506 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:42.677856 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:42.677827 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" path="/var/lib/kubelet/pods/753bcdc7-60a4-4d92-9ca4-e11a33d4e339/volumes" Nov 03 20:34:42.681882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:42.681847 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:44.261129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:44.261059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.814743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:45.814694 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:47.306373 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:47.306332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:47.883428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:47.883389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:48.914359 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:48.914322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:50.569147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:50.569111 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.094214 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:52.094180 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.419421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:53.419381 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.616658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:53.616624 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.925516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:53.925483 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:54.036059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:54.036008 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:55.396486 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:55.396448 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:56.940834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:56.940797 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:58.529281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:34:58.529245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:00.092116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:00.092063 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:01.589281 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:01.589232 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:02.122118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:02.122066 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn"] Nov 03 20:35:02.122375 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:02.122328 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" containerID="cri-o://168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe" gracePeriod=30 Nov 03 20:35:03.198500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:03.198465 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:04.924197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:04.924158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.209025 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:05.208939 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:05.209210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:05.209017 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:05.630226 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:35:05.630192 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:35:06.521249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:06.521204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:07.169662 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:35:07.169632 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff33112_7b0e_445f_a7c2_d8766dfa2d49.slice/crio-conmon-168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:35:07.402535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.402469 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:35:07.545346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545312 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545369 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545453 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545477 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545513 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545541 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545582 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545740 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.545819 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.545768 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo\") pod \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\" (UID: \"cff33112-7b0e-445f-a7c2-d8766dfa2d49\") " Nov 03 20:35:07.546286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546118 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:07.546286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546171 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:07.546286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546207 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:07.546286 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546280 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:07.546500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546383 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:07.546649 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546629 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.546725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546652 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.546725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546665 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.546725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546678 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.546725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.546691 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.548212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.548187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:07.548326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.548224 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "kube-api-access-tk2qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:07.548326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.548260 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:07.548497 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.548404 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo" pod "cff33112-7b0e-445f-a7c2-d8766dfa2d49" (UID: "cff33112-7b0e-445f-a7c2-d8766dfa2d49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:07.647576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.647538 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2qj\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-kube-api-access-tk2qj\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.647576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.647574 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.647576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.647586 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:07.647850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:07.647594 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff33112-7b0e-445f-a7c2-d8766dfa2d49-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:35:08.139369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.139337 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:08.179827 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.179792 5043 generic.go:356] "Generic (PLEG): container finished" podID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerID="168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe" exitCode=0 Nov 03 20:35:08.180033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.179844 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" Nov 03 20:35:08.180033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.179851 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" event={"ID":"cff33112-7b0e-445f-a7c2-d8766dfa2d49","Type":"ContainerDied","Data":"168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe"} Nov 03 20:35:08.180033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.179905 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn" event={"ID":"cff33112-7b0e-445f-a7c2-d8766dfa2d49","Type":"ContainerDied","Data":"8334451728eb039fb427894e65d6dd710d538ad5e1463b279daf5287e7035ac6"} Nov 03 20:35:08.180033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.179933 5043 scope.go:122] "RemoveContainer" containerID="168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe" Nov 03 20:35:08.197251 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.197230 5043 scope.go:122] "RemoveContainer" containerID="168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe" Nov 03 20:35:08.197600 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:35:08.197575 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe\": container with ID starting with 168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe not found: ID does not exist" containerID="168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe" Nov 03 20:35:08.197662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.197611 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe"} err="failed to get container status \"168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe\": rpc error: code = NotFound desc = could not find container \"168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe\": container with ID starting with 168356002aa9d351b6758649e1f3d6540c1b8983b3be28e95a704140c5de0afe not found: ID does not exist" Nov 03 20:35:08.250272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.250245 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn"] Nov 03 20:35:08.255855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.255832 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-t2zhn"] Nov 03 20:35:08.677283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:08.677254 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" path="/var/lib/kubelet/pods/cff33112-7b0e-445f-a7c2-d8766dfa2d49/volumes" Nov 03 20:35:09.920163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:09.920109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.634054 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:11.631176 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:13.204604 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:13.204564 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:14.933876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:14.933834 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:16.428675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:16.428639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:18.015431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:18.015392 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:19.566996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:19.566934 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.883564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:20.883517 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.983842 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:20.983808 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:21.235395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:21.235307 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:22.813018 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:22.812981 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:24.546212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:24.546175 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.011487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:26.011448 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:27.610128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:27.610057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.188284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:29.188246 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:30.817138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:30.817064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.181765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:32.181727 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.534890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:32.534771 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:34.100613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:34.100581 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.583726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:35.583681 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.628914 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:35:35.628878 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:35:35.686361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:35.686327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:37.203453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:37.203409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.805387 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:38.805351 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:40.530013 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:40.529965 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:42.203295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:42.203250 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:43.819386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:43.819340 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:45.412987 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:45.412952 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:46.021459 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:46.021418 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:47.005315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:47.005275 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:48.496845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:48.496811 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:49.961503 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:49.961464 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:51.598990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:51.598899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:53.188914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:53.188883 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:54.803295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:54.803259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.299042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:56.299005 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.810468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:57.810430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:59.304915 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:35:59.304878 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:00.961044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:00.960998 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:02.614934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:02.614893 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:04.207357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:04.207322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:05.630334 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:36:05.630295 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:36:05.804284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:05.804244 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:07.408295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:07.408251 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:08.897521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:08.897484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:10.497646 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:10.497610 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.094552 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:12.094511 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.848743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:12.848693 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.803257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:13.802303 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.292927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:15.292884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:16.889885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:16.889851 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:18.317306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:18.317268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:19.427162 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:19.427128 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp"] Nov 03 20:36:19.427620 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:19.427324 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" containerID="cri-o://efda12ec093d2b353b2b0099daa453495592a8a4f905662113b655646b51929a" gracePeriod=30 Nov 03 20:36:20.002506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:20.002473 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:36:20.002668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:20.002533 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:36:20.027860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:20.027815 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:21.611690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:21.611655 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:23.202442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:23.202408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:24.730808 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.730771 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:24.735930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.735903 5043 generic.go:356] "Generic (PLEG): container finished" podID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerID="efda12ec093d2b353b2b0099daa453495592a8a4f905662113b655646b51929a" exitCode=0 Nov 03 20:36:24.736097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.735946 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" event={"ID":"e535b50e-9c2f-470d-b676-5c9983c4ea65","Type":"ContainerDied","Data":"efda12ec093d2b353b2b0099daa453495592a8a4f905662113b655646b51929a"} Nov 03 20:36:24.736097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.735988 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" event={"ID":"e535b50e-9c2f-470d-b676-5c9983c4ea65","Type":"ContainerDied","Data":"1d808b3c55f8c35c6c0ae700252718b26ca98722bfc66583d6ec80937ba262d1"} Nov 03 20:36:24.736097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.736004 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d808b3c55f8c35c6c0ae700252718b26ca98722bfc66583d6ec80937ba262d1" Nov 03 20:36:24.736393 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.736380 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:36:24.832581 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.832542 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:24.853598 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.853568 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.853841 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.853822 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.853951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.853864 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854016 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.853952 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854016 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.853990 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.854022 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854147 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.854060 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.854193 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854307 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.854277 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo\") pod \"e535b50e-9c2f-470d-b676-5c9983c4ea65\" (UID: \"e535b50e-9c2f-470d-b676-5c9983c4ea65\") " Nov 03 20:36:24.854650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.854624 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:24.855052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.855024 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:24.855229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.855213 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:24.855468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.855453 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:24.855666 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.855648 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:24.857667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.857646 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:24.857766 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.857664 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:24.858042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.858023 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "kube-api-access-5h5nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:24.858906 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.858886 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token" pod "e535b50e-9c2f-470d-b676-5c9983c4ea65" (UID: "e535b50e-9c2f-470d-b676-5c9983c4ea65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:24.955780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955700 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.955780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955736 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.955780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955751 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e535b50e-9c2f-470d-b676-5c9983c4ea65-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.955780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955765 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.955780 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955779 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.956088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955792 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.956088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955805 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/e535b50e-9c2f-470d-b676-5c9983c4ea65-kube-api-access-5h5nx\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.956088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955820 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e535b50e-9c2f-470d-b676-5c9983c4ea65-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:24.956088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:24.955834 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e535b50e-9c2f-470d-b676-5c9983c4ea65-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:36:25.743358 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:25.743325 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp" Nov 03 20:36:25.805515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:25.805480 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp"] Nov 03 20:36:25.818728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:25.818696 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-k9bcp"] Nov 03 20:36:26.423426 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:26.423354 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:26.678747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:26.678673 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" path="/var/lib/kubelet/pods/e535b50e-9c2f-470d-b676-5c9983c4ea65/volumes" Nov 03 20:36:27.905449 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:27.905410 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:29.593783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:29.593749 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:31.108116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:31.108046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:32.294315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:32.294280 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:32.598268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:32.598230 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:34.199398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:34.199354 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:35.543564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:35.543529 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:35.630247 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:36:35.630203 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:36:35.643920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:35.643874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:36.139503 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:36.139460 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:36.251052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:36.251012 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:37.790725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:37.790676 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:38.267863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:38.267815 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:38.374240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:38.374204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:38.897140 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:38.897106 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:39.522243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:39.522198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:39.652159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:39.652127 5043 scope.go:122] "RemoveContainer" containerID="efda12ec093d2b353b2b0099daa453495592a8a4f905662113b655646b51929a" Nov 03 20:36:41.079735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:41.079689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:42.710999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:42.710958 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:44.194843 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:44.194809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:45.641318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:45.641280 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:45.736751 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:45.736711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:45.837534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:45.837495 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:45.997244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:45.997161 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:47.551211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:47.551167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:47.691675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:47.691640 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:49.362983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:49.362946 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:51.005852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:51.005812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:52.508371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:52.508327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:54.109752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:54.109695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:55.789473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:55.789437 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:57.170833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:57.170799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:57.279754 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:57.279718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:57.794636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:57.794600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.307744 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:58.307707 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.406982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:58.406943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.506609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:58.506573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.936656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:58.936606 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:59.622230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:36:59.622197 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:01.105908 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:01.105872 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:02.685368 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:02.685334 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:03.993116 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:03.993053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:04.482515 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:04.482478 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:04.583670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:04.583629 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:04.683938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:04.683894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:05.175742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:05.175703 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:05.629882 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:37:05.629845 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:37:05.681529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:05.681495 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:05.787658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:05.787624 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:05.996407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:05.996329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:06.440933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:06.440899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:08.016278 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:08.016241 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:09.582441 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:09.582408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:11.161444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:11.161407 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:11.263474 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:11.263432 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:11.477034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:11.476950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:12.984649 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:12.984613 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:14.571507 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:14.571471 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:14.772871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:14.772835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:14.873532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:14.873497 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:16.101720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:16.101671 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:16.994752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:16.994706 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw"] Nov 03 20:37:17.025211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.025175 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6"] Nov 03 20:37:17.132432 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.132394 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.240917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.240881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.323909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.323868 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w"] Nov 03 20:37:17.324406 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.324368 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" podUID="95b70903-bf7e-469e-8542-78c3711b65ec" containerName="maas-api" containerID="cri-o://132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a" gracePeriod=30 Nov 03 20:37:17.379899 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.379863 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.610627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.610592 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:37:17.772803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.772773 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs\") pod \"95b70903-bf7e-469e-8542-78c3711b65ec\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " Nov 03 20:37:17.773005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.772877 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls\") pod \"95b70903-bf7e-469e-8542-78c3711b65ec\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " Nov 03 20:37:17.773005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.772969 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z\" (UniqueName: \"kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z\") pod \"95b70903-bf7e-469e-8542-78c3711b65ec\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " Nov 03 20:37:17.773163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.773121 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca\") pod \"95b70903-bf7e-469e-8542-78c3711b65ec\" (UID: \"95b70903-bf7e-469e-8542-78c3711b65ec\") " Nov 03 20:37:17.773470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.773440 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca" pod "95b70903-bf7e-469e-8542-78c3711b65ec" (UID: "95b70903-bf7e-469e-8542-78c3711b65ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:17.775117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.775060 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs" pod "95b70903-bf7e-469e-8542-78c3711b65ec" (UID: "95b70903-bf7e-469e-8542-78c3711b65ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:17.775346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.775321 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls" pod "95b70903-bf7e-469e-8542-78c3711b65ec" (UID: "95b70903-bf7e-469e-8542-78c3711b65ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:17.776191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.776175 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z" pod "95b70903-bf7e-469e-8542-78c3711b65ec" (UID: "95b70903-bf7e-469e-8542-78c3711b65ec"). InnerVolumeSpecName "kube-api-access-rbd4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:17.874565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.874489 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:17.874565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.874518 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbd4z\" (UniqueName: \"kubernetes.io/projected/95b70903-bf7e-469e-8542-78c3711b65ec-kube-api-access-rbd4z\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:17.874565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.874528 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b70903-bf7e-469e-8542-78c3711b65ec-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:17.874565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:17.874537 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b70903-bf7e-469e-8542-78c3711b65ec-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:18.007144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.007066 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:18.107750 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.107711 5043 generic.go:356] "Generic (PLEG): container finished" podID="95b70903-bf7e-469e-8542-78c3711b65ec" containerID="132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a" exitCode=0 Nov 03 20:37:18.107960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.107768 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" Nov 03 20:37:18.107960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.107792 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" event={"ID":"95b70903-bf7e-469e-8542-78c3711b65ec","Type":"ContainerDied","Data":"132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a"} Nov 03 20:37:18.107960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.107856 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w" event={"ID":"95b70903-bf7e-469e-8542-78c3711b65ec","Type":"ContainerDied","Data":"6158ff269d71c7246a4ff1bf876ef361e73fcf6c3a2982896db252b6b40cac47"} Nov 03 20:37:18.107960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.107886 5043 scope.go:122] "RemoveContainer" containerID="132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a" Nov 03 20:37:18.119634 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.119616 5043 scope.go:122] "RemoveContainer" containerID="132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a" Nov 03 20:37:18.119910 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:37:18.119886 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a\": container with ID starting with 132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a not found: ID does not exist" containerID="132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a" Nov 03 20:37:18.119983 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.119923 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a"} err="failed to get container status \"132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a\": rpc error: code = NotFound desc = could not find container \"132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a\": container with ID starting with 132eb2d9ce2115ca159fb3978448870c38529fa6fce393b135b1ea02ba51e09a not found: ID does not exist" Nov 03 20:37:18.157444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.157386 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w"] Nov 03 20:37:18.169189 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.169168 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-503c4e-7b4fd759f6-khn6w"] Nov 03 20:37:18.348887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.348853 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:18.450555 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.450468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:18.677795 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.677765 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b70903-bf7e-469e-8542-78c3711b65ec" path="/var/lib/kubelet/pods/95b70903-bf7e-469e-8542-78c3711b65ec/volumes" Nov 03 20:37:18.938993 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:18.938958 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:19.838886 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:19.838851 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.402170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:21.402048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.995673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:21.995626 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" podUID="a2e4fe77-f913-44d7-bc46-276176b1c65b" containerName="payload-processing" containerID="cri-o://daf8e9147736c098f86f7691ef1aaa252a7019b7595b0ef0105ba2589829a341" gracePeriod=25 Nov 03 20:37:22.025626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.025584 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" podUID="5ea681b3-4632-45b8-80d4-333939e99c24" containerName="payload-pre-processing" containerID="cri-o://71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10" gracePeriod=25 Nov 03 20:37:22.136878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.136840 5043 generic.go:356] "Generic (PLEG): container finished" podID="a2e4fe77-f913-44d7-bc46-276176b1c65b" containerID="daf8e9147736c098f86f7691ef1aaa252a7019b7595b0ef0105ba2589829a341" exitCode=0 Nov 03 20:37:22.137146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.137115 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" event={"ID":"a2e4fe77-f913-44d7-bc46-276176b1c65b","Type":"ContainerDied","Data":"daf8e9147736c098f86f7691ef1aaa252a7019b7595b0ef0105ba2589829a341"} Nov 03 20:37:22.310221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.310187 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:37:22.353927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.350204 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:37:22.434476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.434429 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62\" (UniqueName: \"kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62\") pod \"a2e4fe77-f913-44d7-bc46-276176b1c65b\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " Nov 03 20:37:22.434891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.434630 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume\") pod \"a2e4fe77-f913-44d7-bc46-276176b1c65b\" (UID: \"a2e4fe77-f913-44d7-bc46-276176b1c65b\") " Nov 03 20:37:22.435081 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.435037 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume" pod "a2e4fe77-f913-44d7-bc46-276176b1c65b" (UID: "a2e4fe77-f913-44d7-bc46-276176b1c65b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:22.436723 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.436696 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62" pod "a2e4fe77-f913-44d7-bc46-276176b1c65b" (UID: "a2e4fe77-f913-44d7-bc46-276176b1c65b"). InnerVolumeSpecName "kube-api-access-7hs62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:22.536154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.536047 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj\" (UniqueName: \"kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj\") pod \"5ea681b3-4632-45b8-80d4-333939e99c24\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " Nov 03 20:37:22.536828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.536800 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume\") pod \"5ea681b3-4632-45b8-80d4-333939e99c24\" (UID: \"5ea681b3-4632-45b8-80d4-333939e99c24\") " Nov 03 20:37:22.537277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.537247 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume" pod "5ea681b3-4632-45b8-80d4-333939e99c24" (UID: "5ea681b3-4632-45b8-80d4-333939e99c24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:22.537504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.537483 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea681b3-4632-45b8-80d4-333939e99c24-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:22.537504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.537506 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hs62\" (UniqueName: \"kubernetes.io/projected/a2e4fe77-f913-44d7-bc46-276176b1c65b-kube-api-access-7hs62\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:22.537686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.537521 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e4fe77-f913-44d7-bc46-276176b1c65b-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:22.539556 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.539518 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj" pod "5ea681b3-4632-45b8-80d4-333939e99c24" (UID: "5ea681b3-4632-45b8-80d4-333939e99c24"). InnerVolumeSpecName "kube-api-access-jvtpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:22.543592 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.543532 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:22.639991 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.639957 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtpj\" (UniqueName: \"kubernetes.io/projected/5ea681b3-4632-45b8-80d4-333939e99c24-kube-api-access-jvtpj\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:22.662693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:22.662654 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:23.002028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.001983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:23.145349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.145318 5043 generic.go:356] "Generic (PLEG): container finished" podID="5ea681b3-4632-45b8-80d4-333939e99c24" containerID="71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10" exitCode=0 Nov 03 20:37:23.145537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.145370 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" Nov 03 20:37:23.145537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.145412 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" event={"ID":"5ea681b3-4632-45b8-80d4-333939e99c24","Type":"ContainerDied","Data":"71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10"} Nov 03 20:37:23.145537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.145462 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6" event={"ID":"5ea681b3-4632-45b8-80d4-333939e99c24","Type":"ContainerDied","Data":"e846dbdd003f4be796288518409db61bd88bb3c6b15e10a5a4f087129da6a812"} Nov 03 20:37:23.145537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.145496 5043 scope.go:122] "RemoveContainer" containerID="71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10" Nov 03 20:37:23.147000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.146981 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" event={"ID":"a2e4fe77-f913-44d7-bc46-276176b1c65b","Type":"ContainerDied","Data":"e2a98c27688072c3cb3d06bd98b5b5c1a8546437744b926d1e1013f8ca93df6f"} Nov 03 20:37:23.147098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.147021 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw" Nov 03 20:37:23.156034 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.156018 5043 scope.go:122] "RemoveContainer" containerID="71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10" Nov 03 20:37:23.156307 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:37:23.156286 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10\": container with ID starting with 71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10 not found: ID does not exist" containerID="71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10" Nov 03 20:37:23.156364 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.156324 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10"} err="failed to get container status \"71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10\": rpc error: code = NotFound desc = could not find container \"71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10\": container with ID starting with 71d6999da5717f75dff8647a9c9331e09e7835c63b4ac113d2946e79cb8f8a10 not found: ID does not exist" Nov 03 20:37:23.156364 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.156349 5043 scope.go:122] "RemoveContainer" containerID="daf8e9147736c098f86f7691ef1aaa252a7019b7595b0ef0105ba2589829a341" Nov 03 20:37:23.201582 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.201541 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6"] Nov 03 20:37:23.207996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.207966 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-503c4e-6df744469f-kbjv6"] Nov 03 20:37:23.238976 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.238943 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw"] Nov 03 20:37:23.244695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.244667 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-503c4e-858f76d468-4dblw"] Nov 03 20:37:23.952803 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:23.952767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:24.058348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:24.058311 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:24.564099 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:24.564038 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:24.677631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:24.677594 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea681b3-4632-45b8-80d4-333939e99c24" path="/var/lib/kubelet/pods/5ea681b3-4632-45b8-80d4-333939e99c24/volumes" Nov 03 20:37:24.678041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:24.678025 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e4fe77-f913-44d7-bc46-276176b1c65b" path="/var/lib/kubelet/pods/a2e4fe77-f913-44d7-bc46-276176b1c65b/volumes" Nov 03 20:37:25.081254 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:25.081207 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:25.188778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:25.188738 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:25.410363 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:25.410276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:26.383229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:26.383183 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:28.013294 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:28.013257 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:29.591981 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:29.591942 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:30.729279 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:30.729244 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz"] Nov 03 20:37:30.730388 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:30.730345 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" podUID="a502b248-8079-481a-8837-f598a2f1829a" containerName="maas-api" containerID="cri-o://87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab" gracePeriod=30 Nov 03 20:37:30.787927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:30.787835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:31.027735 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.027708 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:37:31.061418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.061389 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls\") pod \"a502b248-8079-481a-8837-f598a2f1829a\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " Nov 03 20:37:31.061536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.061430 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs\") pod \"a502b248-8079-481a-8837-f598a2f1829a\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " Nov 03 20:37:31.061593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.061564 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx\" (UniqueName: \"kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx\") pod \"a502b248-8079-481a-8837-f598a2f1829a\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " Nov 03 20:37:31.061674 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.061658 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca\") pod \"a502b248-8079-481a-8837-f598a2f1829a\" (UID: \"a502b248-8079-481a-8837-f598a2f1829a\") " Nov 03 20:37:31.062088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.062048 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca" pod "a502b248-8079-481a-8837-f598a2f1829a" (UID: "a502b248-8079-481a-8837-f598a2f1829a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:31.062248 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.062228 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502b248-8079-481a-8837-f598a2f1829a-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:31.063573 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.063543 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls" pod "a502b248-8079-481a-8837-f598a2f1829a" (UID: "a502b248-8079-481a-8837-f598a2f1829a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:31.063720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.063605 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs" pod "a502b248-8079-481a-8837-f598a2f1829a" (UID: "a502b248-8079-481a-8837-f598a2f1829a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:31.063720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.063694 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx" pod "a502b248-8079-481a-8837-f598a2f1829a" (UID: "a502b248-8079-481a-8837-f598a2f1829a"). InnerVolumeSpecName "kube-api-access-5nffx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:31.163412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.163377 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:31.163412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.163408 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502b248-8079-481a-8837-f598a2f1829a-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:31.163412 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.163419 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nffx\" (UniqueName: \"kubernetes.io/projected/a502b248-8079-481a-8837-f598a2f1829a-kube-api-access-5nffx\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:31.227630 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.227588 5043 generic.go:356] "Generic (PLEG): container finished" podID="a502b248-8079-481a-8837-f598a2f1829a" containerID="87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab" exitCode=0 Nov 03 20:37:31.227837 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.227751 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" event={"ID":"a502b248-8079-481a-8837-f598a2f1829a","Type":"ContainerDied","Data":"87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab"} Nov 03 20:37:31.227910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.227849 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" event={"ID":"a502b248-8079-481a-8837-f598a2f1829a","Type":"ContainerDied","Data":"354ea3708d86162e56af40f6327107022cb804b5689bdfad257978c3d6ab15c6"} Nov 03 20:37:31.227910 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.227878 5043 scope.go:122] "RemoveContainer" containerID="87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab" Nov 03 20:37:31.228062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.228044 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz" Nov 03 20:37:31.251792 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.251763 5043 scope.go:122] "RemoveContainer" containerID="87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab" Nov 03 20:37:31.252152 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:37:31.252127 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab\": container with ID starting with 87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab not found: ID does not exist" containerID="87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab" Nov 03 20:37:31.252229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.252161 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab"} err="failed to get container status \"87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab\": rpc error: code = NotFound desc = could not find container \"87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab\": container with ID starting with 87f7b0837f46d0cdc50e31f49251e8725de7f9c3e6b66180190a5a7131fab2ab not found: ID does not exist" Nov 03 20:37:31.286324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.286297 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz"] Nov 03 20:37:31.290128 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.290102 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b95f3-9b55d9d64-d9lgz"] Nov 03 20:37:31.369186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.369147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:31.469723 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.469687 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:31.569504 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.569412 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:31.923965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:31.923924 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:32.415757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:32.415717 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:32.677782 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:32.677694 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a502b248-8079-481a-8837-f598a2f1829a" path="/var/lib/kubelet/pods/a502b248-8079-481a-8837-f598a2f1829a/volumes" Nov 03 20:37:33.210315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:33.210282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:34.706324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:34.706293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:35.632653 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:37:35.632618 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:37:36.317463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:36.317428 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:36.524465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:36.524427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:36.621613 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:36.621395 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:37.791462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:37.791423 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:37.895017 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:37.894982 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:37.995658 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:37.995619 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:38.128377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:38.128335 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:38.368158 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:38.368118 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:39.553764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:39.553726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:41.216468 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:41.216430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:42.823417 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:42.823381 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.039793 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.039754 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w"] Nov 03 20:37:44.040361 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.040012 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" podUID="8971f511-2a1e-43d9-86bc-2b77098d9982" containerName="maas-api" containerID="cri-o://caee8d91e1db875b5a5d0b8b2c366a36774fe290dde10462333876d763c2a6d5" gracePeriod=30 Nov 03 20:37:44.182233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.182189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.322730 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.322692 5043 generic.go:356] "Generic (PLEG): container finished" podID="8971f511-2a1e-43d9-86bc-2b77098d9982" containerID="caee8d91e1db875b5a5d0b8b2c366a36774fe290dde10462333876d763c2a6d5" exitCode=0 Nov 03 20:37:44.322883 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.322784 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" event={"ID":"8971f511-2a1e-43d9-86bc-2b77098d9982","Type":"ContainerDied","Data":"caee8d91e1db875b5a5d0b8b2c366a36774fe290dde10462333876d763c2a6d5"} Nov 03 20:37:44.345692 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.345669 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:37:44.429212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.429172 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\") pod \"8971f511-2a1e-43d9-86bc-2b77098d9982\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " Nov 03 20:37:44.429423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.429230 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca\") pod \"8971f511-2a1e-43d9-86bc-2b77098d9982\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " Nov 03 20:37:44.429423 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.429328 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs\") pod \"8971f511-2a1e-43d9-86bc-2b77098d9982\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " Nov 03 20:37:44.429567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.429467 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4\" (UniqueName: \"kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4\") pod \"8971f511-2a1e-43d9-86bc-2b77098d9982\" (UID: \"8971f511-2a1e-43d9-86bc-2b77098d9982\") " Nov 03 20:37:44.429930 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.429861 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca" pod "8971f511-2a1e-43d9-86bc-2b77098d9982" (UID: "8971f511-2a1e-43d9-86bc-2b77098d9982"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:44.434265 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.434233 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4" pod "8971f511-2a1e-43d9-86bc-2b77098d9982" (UID: "8971f511-2a1e-43d9-86bc-2b77098d9982"). InnerVolumeSpecName "kube-api-access-2w8j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:44.434817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.434788 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs" pod "8971f511-2a1e-43d9-86bc-2b77098d9982" (UID: "8971f511-2a1e-43d9-86bc-2b77098d9982"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:44.436045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.436027 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls" pod "8971f511-2a1e-43d9-86bc-2b77098d9982" (UID: "8971f511-2a1e-43d9-86bc-2b77098d9982"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:44.532122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.531787 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w8j4\" (UniqueName: \"kubernetes.io/projected/8971f511-2a1e-43d9-86bc-2b77098d9982-kube-api-access-2w8j4\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:44.532122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.531833 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:44.532122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.531848 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8971f511-2a1e-43d9-86bc-2b77098d9982-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:44.532122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.531861 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8971f511-2a1e-43d9-86bc-2b77098d9982-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:37:44.615662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.615621 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.729381 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.729343 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.829903 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:44.829858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:45.253771 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:45.253731 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:45.330143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:45.330110 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" Nov 03 20:37:45.330324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:45.330108 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w" event={"ID":"8971f511-2a1e-43d9-86bc-2b77098d9982","Type":"ContainerDied","Data":"d300cd55d98772c7d74f200c84f2b36621671a600cfe00f938d40d5be1d28e22"} Nov 03 20:37:45.330324 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:45.330287 5043 scope.go:122] "RemoveContainer" containerID="caee8d91e1db875b5a5d0b8b2c366a36774fe290dde10462333876d763c2a6d5" Nov 03 20:37:45.381611 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:45.381575 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w"] Nov 03 20:37:45.393637 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:45.393604 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a0866c-6566cb8b5-nqv2w"] Nov 03 20:37:46.603301 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:46.603266 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:46.679184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:46.679150 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8971f511-2a1e-43d9-86bc-2b77098d9982" path="/var/lib/kubelet/pods/8971f511-2a1e-43d9-86bc-2b77098d9982/volumes" Nov 03 20:37:48.203120 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:48.203048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:49.341221 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:49.338986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:49.446578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:49.446542 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:49.537226 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:49.537180 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:50.142937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:50.142899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:50.554588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:50.554496 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:50.655334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:50.655297 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:51.162088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:51.162027 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:51.532681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:51.532585 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:51.641941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:51.641904 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:51.875467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:51.875436 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:51.969596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:51.969562 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:52.335022 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:52.334981 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:53.629193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:53.629156 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:54.297931 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:54.297889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:54.402059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:54.402020 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:55.302121 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:55.302081 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:56.820615 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:56.820573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:57.431392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:57.431353 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:57.606322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:57.606284 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:57.708129 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:57.708039 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:57.937550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:57.937516 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:58.422399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:37:58.422356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:00.208384 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:00.208336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:01.709846 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:01.709809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:02.973626 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:02.973591 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7"] Nov 03 20:38:03.114000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:03.113965 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:03.203513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:03.203477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:03.430007 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:03.429973 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:05.007100 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:05.007056 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:05.629284 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:05.629247 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:38:06.621828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:06.621788 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:07.974917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:07.974877 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" podUID="0ba0abe0-719e-4208-a2d6-790a5c0dec00" containerName="payload-processing" containerID="cri-o://615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd" gracePeriod=25 Nov 03 20:38:08.212755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.212719 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:08.237675 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.237616 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:38:08.328369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.328341 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume\") pod \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " Nov 03 20:38:08.328578 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.328558 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq\" (UniqueName: \"kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq\") pod \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\" (UID: \"0ba0abe0-719e-4208-a2d6-790a5c0dec00\") " Nov 03 20:38:08.328695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.328673 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume" pod "0ba0abe0-719e-4208-a2d6-790a5c0dec00" (UID: "0ba0abe0-719e-4208-a2d6-790a5c0dec00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:08.328922 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.328906 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ba0abe0-719e-4208-a2d6-790a5c0dec00-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:08.331112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.331079 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq" pod "0ba0abe0-719e-4208-a2d6-790a5c0dec00" (UID: "0ba0abe0-719e-4208-a2d6-790a5c0dec00"). InnerVolumeSpecName "kube-api-access-5xnnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:08.429844 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.429815 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnnq\" (UniqueName: \"kubernetes.io/projected/0ba0abe0-719e-4208-a2d6-790a5c0dec00-kube-api-access-5xnnq\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:08.478223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.478183 5043 generic.go:356] "Generic (PLEG): container finished" podID="0ba0abe0-719e-4208-a2d6-790a5c0dec00" containerID="615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd" exitCode=0 Nov 03 20:38:08.478397 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.478235 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" Nov 03 20:38:08.478397 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.478226 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" event={"ID":"0ba0abe0-719e-4208-a2d6-790a5c0dec00","Type":"ContainerDied","Data":"615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd"} Nov 03 20:38:08.478397 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.478366 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7" event={"ID":"0ba0abe0-719e-4208-a2d6-790a5c0dec00","Type":"ContainerDied","Data":"d05276a9cfb62fbbb7519374cd1e0cb97c73ece637b6713aeaa43a9a60b07dda"} Nov 03 20:38:08.478397 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.478396 5043 scope.go:122] "RemoveContainer" containerID="615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd" Nov 03 20:38:08.488945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.488928 5043 scope.go:122] "RemoveContainer" containerID="615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd" Nov 03 20:38:08.489248 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:08.489229 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd\": container with ID starting with 615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd not found: ID does not exist" containerID="615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd" Nov 03 20:38:08.489333 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.489252 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd"} err="failed to get container status \"615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd\": rpc error: code = NotFound desc = could not find container \"615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd\": container with ID starting with 615ca9d51ad8d35f7dbe4cad3202055a932366c06881fad0b6a0d57211701bdd not found: ID does not exist" Nov 03 20:38:08.531746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.531718 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7"] Nov 03 20:38:08.536882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.536860 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e80203-5dfc9556d6-98js7"] Nov 03 20:38:08.676961 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.676924 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba0abe0-719e-4208-a2d6-790a5c0dec00" path="/var/lib/kubelet/pods/0ba0abe0-719e-4208-a2d6-790a5c0dec00/volumes" Nov 03 20:38:08.906197 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:08.906168 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5"] Nov 03 20:38:09.013580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:09.013543 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:09.115215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:09.115180 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:09.684937 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:09.684902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:10.342340 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:10.342297 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:11.910349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:11.910315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:13.697757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:13.697715 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:13.797992 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:13.797949 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:13.898133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:13.898099 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:13.906817 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:13.906787 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" podUID="438055de-8964-4c06-a10b-786b96c44d8b" containerName="payload-pre-processing" containerID="cri-o://931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c" gracePeriod=25 Nov 03 20:38:14.189777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.189746 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:38:14.214557 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.214525 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68\" (UniqueName: \"kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68\") pod \"438055de-8964-4c06-a10b-786b96c44d8b\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " Nov 03 20:38:14.214726 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.214709 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume\") pod \"438055de-8964-4c06-a10b-786b96c44d8b\" (UID: \"438055de-8964-4c06-a10b-786b96c44d8b\") " Nov 03 20:38:14.215184 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.215151 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume" pod "438055de-8964-4c06-a10b-786b96c44d8b" (UID: "438055de-8964-4c06-a10b-786b96c44d8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:14.216941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.216900 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68" pod "438055de-8964-4c06-a10b-786b96c44d8b" (UID: "438055de-8964-4c06-a10b-786b96c44d8b"). InnerVolumeSpecName "kube-api-access-q5f68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:14.268805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.268722 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:14.316183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.316110 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/438055de-8964-4c06-a10b-786b96c44d8b-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:14.316183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.316151 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5f68\" (UniqueName: \"kubernetes.io/projected/438055de-8964-4c06-a10b-786b96c44d8b-kube-api-access-q5f68\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:14.548984 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.548950 5043 generic.go:356] "Generic (PLEG): container finished" podID="438055de-8964-4c06-a10b-786b96c44d8b" containerID="931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c" exitCode=0 Nov 03 20:38:14.549183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.548998 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" Nov 03 20:38:14.549183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.549024 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" event={"ID":"438055de-8964-4c06-a10b-786b96c44d8b","Type":"ContainerDied","Data":"931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c"} Nov 03 20:38:14.549183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.549102 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5" event={"ID":"438055de-8964-4c06-a10b-786b96c44d8b","Type":"ContainerDied","Data":"dad638ab2bf953f9a2c9096da0aa2baa39465138df6214a4ce7d51cfde65e2b7"} Nov 03 20:38:14.549183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.549131 5043 scope.go:122] "RemoveContainer" containerID="931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c" Nov 03 20:38:14.559572 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.559540 5043 scope.go:122] "RemoveContainer" containerID="931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c" Nov 03 20:38:14.559820 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:14.559799 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c\": container with ID starting with 931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c not found: ID does not exist" containerID="931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c" Nov 03 20:38:14.559896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.559832 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c"} err="failed to get container status \"931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c\": rpc error: code = NotFound desc = could not find container \"931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c\": container with ID starting with 931882176e375798d082924b6ab4b7bac5ed30c563239018fdc834c04a0a100c not found: ID does not exist" Nov 03 20:38:14.598919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.598889 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5"] Nov 03 20:38:14.610723 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.610698 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d1bfb8-cc86454f5-fbnh5"] Nov 03 20:38:14.678490 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:14.678454 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438055de-8964-4c06-a10b-786b96c44d8b" path="/var/lib/kubelet/pods/438055de-8964-4c06-a10b-786b96c44d8b/volumes" Nov 03 20:38:15.257051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.257019 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc"] Nov 03 20:38:15.257575 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.257303 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" podUID="b5f72b17-0904-4497-8705-32c31f8b3b4d" containerName="maas-api" containerID="cri-o://b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862" gracePeriod=30 Nov 03 20:38:15.329035 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.329003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:15.552958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.552932 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:38:15.557798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.557770 5043 generic.go:356] "Generic (PLEG): container finished" podID="b5f72b17-0904-4497-8705-32c31f8b3b4d" containerID="b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862" exitCode=0 Nov 03 20:38:15.557933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.557810 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" Nov 03 20:38:15.557933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.557814 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" event={"ID":"b5f72b17-0904-4497-8705-32c31f8b3b4d","Type":"ContainerDied","Data":"b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862"} Nov 03 20:38:15.558039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.557956 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc" event={"ID":"b5f72b17-0904-4497-8705-32c31f8b3b4d","Type":"ContainerDied","Data":"b05958a90c625d363b4c2cf97e6f02f7e4b5aaa0623d7bd2dcc0fcfaeced881c"} Nov 03 20:38:15.558039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.557985 5043 scope.go:122] "RemoveContainer" containerID="b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862" Nov 03 20:38:15.568824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.568804 5043 scope.go:122] "RemoveContainer" containerID="b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862" Nov 03 20:38:15.569131 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:15.569104 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862\": container with ID starting with b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862 not found: ID does not exist" containerID="b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862" Nov 03 20:38:15.569270 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.569140 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862"} err="failed to get container status \"b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862\": rpc error: code = NotFound desc = could not find container \"b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862\": container with ID starting with b14ae56e5952f6da59cbb56d9b11993627be968cb5d25ea94f92adb6fb634862 not found: ID does not exist" Nov 03 20:38:15.635357 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.635319 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59\" (UniqueName: \"kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59\") pod \"b5f72b17-0904-4497-8705-32c31f8b3b4d\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " Nov 03 20:38:15.635565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.635419 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca\") pod \"b5f72b17-0904-4497-8705-32c31f8b3b4d\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " Nov 03 20:38:15.635565 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.635498 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls\") pod \"b5f72b17-0904-4497-8705-32c31f8b3b4d\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " Nov 03 20:38:15.635714 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.635615 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs\") pod \"b5f72b17-0904-4497-8705-32c31f8b3b4d\" (UID: \"b5f72b17-0904-4497-8705-32c31f8b3b4d\") " Nov 03 20:38:15.636088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.636040 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca" pod "b5f72b17-0904-4497-8705-32c31f8b3b4d" (UID: "b5f72b17-0904-4497-8705-32c31f8b3b4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:15.639677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.639639 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59" pod "b5f72b17-0904-4497-8705-32c31f8b3b4d" (UID: "b5f72b17-0904-4497-8705-32c31f8b3b4d"). InnerVolumeSpecName "kube-api-access-tgv59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:15.639833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.639808 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs" pod "b5f72b17-0904-4497-8705-32c31f8b3b4d" (UID: "b5f72b17-0904-4497-8705-32c31f8b3b4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:15.643342 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.643317 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls" pod "b5f72b17-0904-4497-8705-32c31f8b3b4d" (UID: "b5f72b17-0904-4497-8705-32c31f8b3b4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:15.741092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.739016 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:15.741092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.739092 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f72b17-0904-4497-8705-32c31f8b3b4d-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:15.741092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.739111 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgv59\" (UniqueName: \"kubernetes.io/projected/b5f72b17-0904-4497-8705-32c31f8b3b4d-kube-api-access-tgv59\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:15.741092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.739126 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f72b17-0904-4497-8705-32c31f8b3b4d-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:15.817875 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.817776 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:15.905407 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.905289 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc"] Nov 03 20:38:15.915125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.915093 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e80203-9c64b7955-xl5rc"] Nov 03 20:38:15.918610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:15.918578 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.034320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:16.034283 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.435338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:16.435303 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.677761 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:16.677725 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f72b17-0904-4497-8705-32c31f8b3b4d" path="/var/lib/kubelet/pods/b5f72b17-0904-4497-8705-32c31f8b3b4d/volumes" Nov 03 20:38:17.709804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:17.709770 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:19.207145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:19.207107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.487482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:20.487446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.587654 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:20.587622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.750820 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:20.750741 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc"] Nov 03 20:38:20.751082 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:20.751043 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" containerID="cri-o://3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a" gracePeriod=30 Nov 03 20:38:20.795443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:20.795412 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.901126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:20.900568 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:21.230549 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:21.230501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:22.619745 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:22.619686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:22.759972 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:22.759930 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:22.890305 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:22.890203 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:23.091132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:23.091090 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59"] Nov 03 20:38:23.091366 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:23.091341 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" containerID="cri-o://81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c" gracePeriod=30 Nov 03 20:38:23.118943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:23.118905 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:24.227378 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:24.227342 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:25.690449 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:25.690406 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:26.314811 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.314774 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:26.432293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.432271 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:38:26.461500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461472 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.461641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461563 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.461700 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461686 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.461756 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461724 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.461798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461759 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.461853 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461835 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.461913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461879 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:26.461977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.461927 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:26.462055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.462036 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:26.462131 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.462106 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:26.462177 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.462143 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.462341 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.462317 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:26.463145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463123 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.463283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463217 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy\") pod \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\" (UID: \"be65ccf6-55cd-426f-8c1a-b677eddeb02f\") " Nov 03 20:38:26.463946 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463926 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.464094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463949 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.464094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463963 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.464094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463976 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.464094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.463989 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.464339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.464309 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "kube-api-access-ztgr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:26.465437 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.465408 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:38:26.465525 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.465481 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:26.466392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.466369 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token" pod "be65ccf6-55cd-426f-8c1a-b677eddeb02f" (UID: "be65ccf6-55cd-426f-8c1a-b677eddeb02f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:26.565276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.565205 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.565276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.565231 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.565276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.565240 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgr7\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-kube-api-access-ztgr7\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.565276 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.565250 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be65ccf6-55cd-426f-8c1a-b677eddeb02f-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:26.633229 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.633194 5043 generic.go:356] "Generic (PLEG): container finished" podID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerID="3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a" exitCode=0 Nov 03 20:38:26.633402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.633243 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" Nov 03 20:38:26.633402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.633249 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" event={"ID":"be65ccf6-55cd-426f-8c1a-b677eddeb02f","Type":"ContainerDied","Data":"3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a"} Nov 03 20:38:26.633402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.633292 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc" event={"ID":"be65ccf6-55cd-426f-8c1a-b677eddeb02f","Type":"ContainerDied","Data":"f3a3074b2d26ece87c3bb38bde736e9c9fff4c552d4059481702b55060471be5"} Nov 03 20:38:26.633402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.633313 5043 scope.go:122] "RemoveContainer" containerID="3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a" Nov 03 20:38:26.646176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.646148 5043 scope.go:122] "RemoveContainer" containerID="2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade" Nov 03 20:38:26.656882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.656858 5043 scope.go:122] "RemoveContainer" containerID="3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a" Nov 03 20:38:26.657175 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:26.657154 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a\": container with ID starting with 3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a not found: ID does not exist" containerID="3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a" Nov 03 20:38:26.657252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.657184 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a"} err="failed to get container status \"3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a\": rpc error: code = NotFound desc = could not find container \"3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a\": container with ID starting with 3b2ffff803d6933b44d6923b3cded309c89dfa1c1ec1ec411cdaca5c4e22ba8a not found: ID does not exist" Nov 03 20:38:26.657252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.657204 5043 scope.go:122] "RemoveContainer" containerID="2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade" Nov 03 20:38:26.657455 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:26.657427 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade\": container with ID starting with 2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade not found: ID does not exist" containerID="2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade" Nov 03 20:38:26.657506 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.657463 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade"} err="failed to get container status \"2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade\": rpc error: code = NotFound desc = could not find container \"2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade\": container with ID starting with 2c71bd9ebbd7f3b25c1cac2f78920b393bd36a1689adf6254acc8d0a9018fade not found: ID does not exist" Nov 03 20:38:26.688670 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.688592 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc"] Nov 03 20:38:26.692420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:26.692391 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d1bfb8-openshift-default-5f85bdc98f-lllbc"] Nov 03 20:38:27.307191 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:27.307155 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:28.366928 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.366863 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:38:28.490522 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490494 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490560 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490592 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490681 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490643 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490695 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490734 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490757 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:28.490864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490805 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.490864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490850 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.491169 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.490920 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert\") pod \"9ae01033-875e-4913-9671-9a2464a837a6\" (UID: \"9ae01033-875e-4913-9671-9a2464a837a6\") " Nov 03 20:38:28.491238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491194 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:28.491238 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491212 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:28.491419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491396 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:28.491419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491410 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.491550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491430 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.491550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491446 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.491550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.491539 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:28.493117 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.493093 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:28.493419 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.493388 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "kube-api-access-c556l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:28.493516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.493491 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:28.493746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.493730 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo" pod "9ae01033-875e-4913-9671-9a2464a837a6" (UID: "9ae01033-875e-4913-9671-9a2464a837a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:38:28.592712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.592675 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae01033-875e-4913-9671-9a2464a837a6-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.592712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.592704 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.592712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.592713 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae01033-875e-4913-9671-9a2464a837a6-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.592913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.592722 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.592913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.592731 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae01033-875e-4913-9671-9a2464a837a6-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.592913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.592740 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c556l\" (UniqueName: \"kubernetes.io/projected/9ae01033-875e-4913-9671-9a2464a837a6-kube-api-access-c556l\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:28.646975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.646888 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ae01033-875e-4913-9671-9a2464a837a6" containerID="81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c" exitCode=0 Nov 03 20:38:28.646975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.646940 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" Nov 03 20:38:28.647249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.646979 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" event={"ID":"9ae01033-875e-4913-9671-9a2464a837a6","Type":"ContainerDied","Data":"81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c"} Nov 03 20:38:28.647249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.647088 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59" event={"ID":"9ae01033-875e-4913-9671-9a2464a837a6","Type":"ContainerDied","Data":"6de22410a47032a3cd1e3b013af68782fd679fca838b1e2f675b0ce0bc3a43c1"} Nov 03 20:38:28.647249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.647125 5043 scope.go:122] "RemoveContainer" containerID="81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c" Nov 03 20:38:28.657822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.657805 5043 scope.go:122] "RemoveContainer" containerID="81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c" Nov 03 20:38:28.658147 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:28.658124 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c\": container with ID starting with 81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c not found: ID does not exist" containerID="81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c" Nov 03 20:38:28.658242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.658153 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c"} err="failed to get container status \"81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c\": rpc error: code = NotFound desc = could not find container \"81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c\": container with ID starting with 81e6eee419fe26593b457ed6ddf4ed6c98a086f42be33cc0a780f03739a3db2c not found: ID does not exist" Nov 03 20:38:28.677314 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.677294 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" path="/var/lib/kubelet/pods/be65ccf6-55cd-426f-8c1a-b677eddeb02f/volumes" Nov 03 20:38:28.697475 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.697444 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59"] Nov 03 20:38:28.708834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.708812 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-bxm59"] Nov 03 20:38:28.909976 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:28.909881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:30.532223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:30.532186 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:30.677182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:30.677144 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae01033-875e-4913-9671-9a2464a837a6" path="/var/lib/kubelet/pods/9ae01033-875e-4913-9671-9a2464a837a6/volumes" Nov 03 20:38:31.874986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:31.874949 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f"] Nov 03 20:38:31.906300 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:31.906265 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j"] Nov 03 20:38:32.005205 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:32.005165 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:32.105759 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:32.105710 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:32.209686 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:32.209596 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:32.534465 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:32.534376 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:33.633036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:33.632990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:33.732979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:33.732902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:35.056836 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:35.056753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:35.629681 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:35.629636 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:38:36.876006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:36.875954 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerName="payload-processing" containerID="cri-o://3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9" gracePeriod=25 Nov 03 20:38:36.906749 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:36.906697 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" containerName="payload-pre-processing" containerID="cri-o://e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1" gracePeriod=25 Nov 03 20:38:37.338614 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:37.338580 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:38:39.220524 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:39.220497 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:38:39.220990 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:39.220940 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:38:39.227588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:39.227567 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:38:39.227919 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:39.227902 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:38:40.609167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:40.609135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:41.782945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:41.782911 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:46.325002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:46.324964 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:38:47.338807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:47.338756 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:38:48.516171 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.516134 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn"] Nov 03 20:38:48.516678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.516445 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" podUID="344ac58a-c539-4a11-85f5-1c285fd570bf" containerName="maas-api" containerID="cri-o://a3a9f3b53391022488968a2a43da87e2c8b310067d983ed50120f44874729d73" gracePeriod=30 Nov 03 20:38:48.573694 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.573664 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:48.790512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.790477 5043 generic.go:356] "Generic (PLEG): container finished" podID="344ac58a-c539-4a11-85f5-1c285fd570bf" containerID="a3a9f3b53391022488968a2a43da87e2c8b310067d983ed50120f44874729d73" exitCode=0 Nov 03 20:38:48.790732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.790714 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" event={"ID":"344ac58a-c539-4a11-85f5-1c285fd570bf","Type":"ContainerDied","Data":"a3a9f3b53391022488968a2a43da87e2c8b310067d983ed50120f44874729d73"} Nov 03 20:38:48.817166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.817139 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:38:48.912808 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.912769 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs\") pod \"344ac58a-c539-4a11-85f5-1c285fd570bf\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " Nov 03 20:38:48.912958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.912851 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca\") pod \"344ac58a-c539-4a11-85f5-1c285fd570bf\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " Nov 03 20:38:48.913021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.912973 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls\") pod \"344ac58a-c539-4a11-85f5-1c285fd570bf\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " Nov 03 20:38:48.913093 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.913045 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm\" (UniqueName: \"kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm\") pod \"344ac58a-c539-4a11-85f5-1c285fd570bf\" (UID: \"344ac58a-c539-4a11-85f5-1c285fd570bf\") " Nov 03 20:38:48.913325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.913290 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca" pod "344ac58a-c539-4a11-85f5-1c285fd570bf" (UID: "344ac58a-c539-4a11-85f5-1c285fd570bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:48.913537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.913497 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344ac58a-c539-4a11-85f5-1c285fd570bf-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:48.915124 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.915099 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs" pod "344ac58a-c539-4a11-85f5-1c285fd570bf" (UID: "344ac58a-c539-4a11-85f5-1c285fd570bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:48.915677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.915647 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls" pod "344ac58a-c539-4a11-85f5-1c285fd570bf" (UID: "344ac58a-c539-4a11-85f5-1c285fd570bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:48.915677 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:48.915665 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm" pod "344ac58a-c539-4a11-85f5-1c285fd570bf" (UID: "344ac58a-c539-4a11-85f5-1c285fd570bf"). InnerVolumeSpecName "kube-api-access-sqhmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:49.014814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.014774 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:49.014814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.014810 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqhmm\" (UniqueName: \"kubernetes.io/projected/344ac58a-c539-4a11-85f5-1c285fd570bf-kube-api-access-sqhmm\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:49.014814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.014822 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344ac58a-c539-4a11-85f5-1c285fd570bf-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:49.034688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.034636 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:49.801952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.801910 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" event={"ID":"344ac58a-c539-4a11-85f5-1c285fd570bf","Type":"ContainerDied","Data":"738f2471120339ed3f19ea2901e4a1ad58579b848414fabd41dd9c7271e1cf51"} Nov 03 20:38:49.802434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.801994 5043 scope.go:122] "RemoveContainer" containerID="a3a9f3b53391022488968a2a43da87e2c8b310067d983ed50120f44874729d73" Nov 03 20:38:49.802434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.802229 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn" Nov 03 20:38:49.818349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.818259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:49.857044 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.857011 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn"] Nov 03 20:38:49.868871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:49.868825 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-482722-67b5f77789-5b6tn"] Nov 03 20:38:50.677421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:50.677386 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344ac58a-c539-4a11-85f5-1c285fd570bf" path="/var/lib/kubelet/pods/344ac58a-c539-4a11-85f5-1c285fd570bf/volumes" Nov 03 20:38:55.309500 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:55.309461 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:55.399631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:55.399596 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:55.587362 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:55.587308 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq"] Nov 03 20:38:55.587678 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:55.587615 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" containerID="cri-o://99cc26dd41e9ef63239980fb4360f12769d50a058409dcced5c750580f7e676f" gracePeriod=30 Nov 03 20:38:55.619425 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:55.619389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:56.325215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:56.325173 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:38:56.736448 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:56.736351 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:56.835917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:56.835883 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:57.186718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.186688 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:38:57.209236 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.209212 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:38:57.305430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.305348 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb\" (UniqueName: \"kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb\") pod \"261cb760-7907-4d77-acf3-2f2cd927d303\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " Nov 03 20:38:57.305430 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.305422 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume\") pod \"0cdb90f0-5685-43db-892b-d7f803178ad1\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " Nov 03 20:38:57.305612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.305479 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume\") pod \"261cb760-7907-4d77-acf3-2f2cd927d303\" (UID: \"261cb760-7907-4d77-acf3-2f2cd927d303\") " Nov 03 20:38:57.305612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.305512 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj\" (UniqueName: \"kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj\") pod \"0cdb90f0-5685-43db-892b-d7f803178ad1\" (UID: \"0cdb90f0-5685-43db-892b-d7f803178ad1\") " Nov 03 20:38:57.305926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.305893 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume" pod "0cdb90f0-5685-43db-892b-d7f803178ad1" (UID: "0cdb90f0-5685-43db-892b-d7f803178ad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:57.305926 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.305919 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume" pod "261cb760-7907-4d77-acf3-2f2cd927d303" (UID: "261cb760-7907-4d77-acf3-2f2cd927d303"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:57.307672 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.307644 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb" pod "261cb760-7907-4d77-acf3-2f2cd927d303" (UID: "261cb760-7907-4d77-acf3-2f2cd927d303"). InnerVolumeSpecName "kube-api-access-zfhlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:57.307882 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.307865 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj" pod "0cdb90f0-5685-43db-892b-d7f803178ad1" (UID: "0cdb90f0-5685-43db-892b-d7f803178ad1"). InnerVolumeSpecName "kube-api-access-5tqbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:57.407416 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.407382 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdb90f0-5685-43db-892b-d7f803178ad1-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:57.407416 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.407410 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261cb760-7907-4d77-acf3-2f2cd927d303-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:57.407416 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.407420 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqbj\" (UniqueName: \"kubernetes.io/projected/0cdb90f0-5685-43db-892b-d7f803178ad1-kube-api-access-5tqbj\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:57.407850 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.407430 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhlb\" (UniqueName: \"kubernetes.io/projected/261cb760-7907-4d77-acf3-2f2cd927d303-kube-api-access-zfhlb\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:38:57.855139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.855103 5043 generic.go:356] "Generic (PLEG): container finished" podID="261cb760-7907-4d77-acf3-2f2cd927d303" containerID="e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1" exitCode=0 Nov 03 20:38:57.855325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.855139 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" Nov 03 20:38:57.855325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.855170 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" event={"ID":"261cb760-7907-4d77-acf3-2f2cd927d303","Type":"ContainerDied","Data":"e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1"} Nov 03 20:38:57.855325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.855228 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j" event={"ID":"261cb760-7907-4d77-acf3-2f2cd927d303","Type":"ContainerDied","Data":"cafccfdf8f40bb9f14a639bdc1faf17dc407e004a53cebd79808627d40fab342"} Nov 03 20:38:57.855325 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.855252 5043 scope.go:122] "RemoveContainer" containerID="e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1" Nov 03 20:38:57.856740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.856714 5043 generic.go:356] "Generic (PLEG): container finished" podID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerID="3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9" exitCode=0 Nov 03 20:38:57.856876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.856796 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" Nov 03 20:38:57.856932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.856795 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" event={"ID":"0cdb90f0-5685-43db-892b-d7f803178ad1","Type":"ContainerDied","Data":"3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9"} Nov 03 20:38:57.856932 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.856927 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f" event={"ID":"0cdb90f0-5685-43db-892b-d7f803178ad1","Type":"ContainerDied","Data":"97262d210418faa544750598fee6b917542b0502d9c8bb0a1408273475e80732"} Nov 03 20:38:57.867641 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.867622 5043 scope.go:122] "RemoveContainer" containerID="e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1" Nov 03 20:38:57.867928 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:57.867905 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1\": container with ID starting with e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1 not found: ID does not exist" containerID="e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1" Nov 03 20:38:57.868047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.867932 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1"} err="failed to get container status \"e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1\": rpc error: code = NotFound desc = could not find container \"e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1\": container with ID starting with e5eeb5dbfc3b15306b0758610e137f68ef252cfc7f369ecd423fa1af60bedcb1 not found: ID does not exist" Nov 03 20:38:57.868047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.867949 5043 scope.go:122] "RemoveContainer" containerID="3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9" Nov 03 20:38:57.877138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.877116 5043 scope.go:122] "RemoveContainer" containerID="3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9" Nov 03 20:38:57.877424 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:38:57.877400 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9\": container with ID starting with 3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9 not found: ID does not exist" containerID="3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9" Nov 03 20:38:57.877516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.877434 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9"} err="failed to get container status \"3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9\": rpc error: code = NotFound desc = could not find container \"3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9\": container with ID starting with 3e6e4f22bc385a6572a033835f6b8cdde6f2dd30f2c1cab92ed9f17a511059c9 not found: ID does not exist" Nov 03 20:38:57.905151 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.905048 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j"] Nov 03 20:38:57.917234 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.917206 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-482722-cd9898ff5-fdt8j"] Nov 03 20:38:57.956032 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.955930 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f"] Nov 03 20:38:57.965118 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:57.965066 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-482722-6d55b58dcd-pgl2f"] Nov 03 20:38:58.062215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:58.062170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:58.298283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:58.298186 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:58.526356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:58.526322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:58.683528 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:58.683489 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" path="/var/lib/kubelet/pods/0cdb90f0-5685-43db-892b-d7f803178ad1/volumes" Nov 03 20:38:58.684105 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:58.684057 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" path="/var/lib/kubelet/pods/261cb760-7907-4d77-acf3-2f2cd927d303/volumes" Nov 03 20:38:58.718815 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:58.718780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:59.953667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:38:59.953629 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:00.365400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.365365 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-7sqwf"] Nov 03 20:39:00.365594 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.365567 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" podUID="b49f54e4-98a5-46cc-b13c-8970fd0cc164" containerName="manager" containerID="cri-o://d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c" gracePeriod=30 Nov 03 20:39:00.708168 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.708146 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:39:00.858228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.858196 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc\" (UniqueName: \"kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc\") pod \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " Nov 03 20:39:00.858392 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.858317 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs\") pod \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " Nov 03 20:39:00.858457 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.858435 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert\") pod \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\" (UID: \"b49f54e4-98a5-46cc-b13c-8970fd0cc164\") " Nov 03 20:39:00.899293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.899115 5043 generic.go:356] "Generic (PLEG): container finished" podID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerID="99cc26dd41e9ef63239980fb4360f12769d50a058409dcced5c750580f7e676f" exitCode=0 Nov 03 20:39:00.899293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.899212 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" event={"ID":"d835a0bf-ebcc-4919-9914-747f20a52fce","Type":"ContainerDied","Data":"99cc26dd41e9ef63239980fb4360f12769d50a058409dcced5c750580f7e676f"} Nov 03 20:39:00.906200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.904024 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs" pod "b49f54e4-98a5-46cc-b13c-8970fd0cc164" (UID: "b49f54e4-98a5-46cc-b13c-8970fd0cc164"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:00.911332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.910562 5043 generic.go:356] "Generic (PLEG): container finished" podID="b49f54e4-98a5-46cc-b13c-8970fd0cc164" containerID="d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c" exitCode=0 Nov 03 20:39:00.911332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.910651 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" event={"ID":"b49f54e4-98a5-46cc-b13c-8970fd0cc164","Type":"ContainerDied","Data":"d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c"} Nov 03 20:39:00.911332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.910705 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" event={"ID":"b49f54e4-98a5-46cc-b13c-8970fd0cc164","Type":"ContainerDied","Data":"d6e8d42fe2ad26c016de0f4265ddbd9f293f64d5f2556ddffb08410082ea09c8"} Nov 03 20:39:00.911332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.910733 5043 scope.go:122] "RemoveContainer" containerID="d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c" Nov 03 20:39:00.911332 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.910932 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-7sqwf" Nov 03 20:39:00.916709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.916656 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc" pod "b49f54e4-98a5-46cc-b13c-8970fd0cc164" (UID: "b49f54e4-98a5-46cc-b13c-8970fd0cc164"). InnerVolumeSpecName "kube-api-access-wpszc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:00.921589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.921548 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert" pod "b49f54e4-98a5-46cc-b13c-8970fd0cc164" (UID: "b49f54e4-98a5-46cc-b13c-8970fd0cc164"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:00.946062 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.946042 5043 scope.go:122] "RemoveContainer" containerID="d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c" Nov 03 20:39:00.947004 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:39:00.946973 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c\": container with ID starting with d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c not found: ID does not exist" containerID="d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c" Nov 03 20:39:00.947156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.947007 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c"} err="failed to get container status \"d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c\": rpc error: code = NotFound desc = could not find container \"d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c\": container with ID starting with d7b4788ed5e5231e6d2f3c9acb9ec8af4293d50f59040d15066a1cbc85cf1d6c not found: ID does not exist" Nov 03 20:39:00.956130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.952845 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:39:00.959767 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.959741 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpszc\" (UniqueName: \"kubernetes.io/projected/b49f54e4-98a5-46cc-b13c-8970fd0cc164-kube-api-access-wpszc\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:00.959767 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.959765 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:00.959944 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:00.959775 5043 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b49f54e4-98a5-46cc-b13c-8970fd0cc164-webhook-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.061056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061002 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061162 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061201 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061284 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061269 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061315 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061350 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061478 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061420 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061505 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.061663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.061546 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket\") pod \"d835a0bf-ebcc-4919-9914-747f20a52fce\" (UID: \"d835a0bf-ebcc-4919-9914-747f20a52fce\") " Nov 03 20:39:01.062228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.062131 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:01.062228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.062057 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:01.062228 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.062182 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:01.062586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.062354 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:01.062586 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.062438 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:01.064267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.064236 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:01.064865 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.064833 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:01.064959 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.064939 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:01.066265 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.066230 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9" pod "d835a0bf-ebcc-4919-9914-747f20a52fce" (UID: "d835a0bf-ebcc-4919-9914-747f20a52fce"). InnerVolumeSpecName "kube-api-access-fszs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162391 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszs9\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-kube-api-access-fszs9\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162421 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-envoy\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162431 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162439 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-data\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162447 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-podinfo\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162454 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-workload-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162466 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d835a0bf-ebcc-4919-9914-747f20a52fce-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162475 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d835a0bf-ebcc-4919-9914-747f20a52fce-credential-socket\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.162487 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.162484 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d835a0bf-ebcc-4919-9914-747f20a52fce-istiod-ca-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:39:01.269440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.269403 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-7sqwf"] Nov 03 20:39:01.274481 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.274454 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-7sqwf"] Nov 03 20:39:01.936975 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.936937 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" event={"ID":"d835a0bf-ebcc-4919-9914-747f20a52fce","Type":"ContainerDied","Data":"d46b7e06180f22b8afdec10625f931abdc655fc1ccd616be1a781bccfd14c33f"} Nov 03 20:39:01.937274 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.937253 5043 scope.go:122] "RemoveContainer" containerID="99cc26dd41e9ef63239980fb4360f12769d50a058409dcced5c750580f7e676f" Nov 03 20:39:01.937541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:01.937523 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq" Nov 03 20:39:02.127096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:02.048373 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq"] Nov 03 20:39:02.127096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:02.053742 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-42lfq"] Nov 03 20:39:02.676326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:02.676291 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49f54e4-98a5-46cc-b13c-8970fd0cc164" path="/var/lib/kubelet/pods/b49f54e4-98a5-46cc-b13c-8970fd0cc164/volumes" Nov 03 20:39:02.676764 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:02.676750 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" path="/var/lib/kubelet/pods/d835a0bf-ebcc-4919-9914-747f20a52fce/volumes" Nov 03 20:39:05.357516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:05.357479 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:05.627850 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:39:05.627762 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:39:32.688505 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:32.688472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:33.026122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:33.025997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:34.787142 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:34.787089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:35.628484 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:39:35.628444 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:39:53.654531 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:53.654491 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:53.776791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:39:53.776752 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:02.402135 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.402098 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj"] Nov 03 20:40:02.402662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.402311 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" podUID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" containerName="discovery" containerID="cri-o://cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec" gracePeriod=30 Nov 03 20:40:02.668240 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.668180 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:40:02.717845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.717817 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.717853 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.717885 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.717944 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.717999 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718051 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.718029 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.718121 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p\") pod \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\" (UID: \"32b2a85c-e89f-4a83-a46e-48895b9ac3dc\") " Nov 03 20:40:02.718377 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.718311 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:02.718489 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.718458 5043 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-ca-configmap\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:02.720899 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.720622 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:02.721107 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.721054 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:02.722471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.722445 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "kube-api-access-nk76p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:02.722609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.722507 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:02.722690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.722620 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:02.723720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.723696 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs" pod "32b2a85c-e89f-4a83-a46e-48895b9ac3dc" (UID: "32b2a85c-e89f-4a83-a46e-48895b9ac3dc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:02.820212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.820109 5043 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-csr-dns-cert\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:02.820212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.820144 5043 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-local-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:02.820212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.820161 5043 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-cacerts\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:02.820212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.820174 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk76p\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-kube-api-access-nk76p\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:02.820212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.820188 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-token\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:02.820212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:02.820200 5043 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32b2a85c-e89f-4a83-a46e-48895b9ac3dc-istio-kubeconfig\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:40:03.299295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.299252 5043 generic.go:356] "Generic (PLEG): container finished" podID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" containerID="cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec" exitCode=0 Nov 03 20:40:03.299740 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.299712 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" event={"ID":"32b2a85c-e89f-4a83-a46e-48895b9ac3dc","Type":"ContainerDied","Data":"cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec"} Nov 03 20:40:03.299943 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.299927 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" event={"ID":"32b2a85c-e89f-4a83-a46e-48895b9ac3dc","Type":"ContainerDied","Data":"964775b3a2bd2c672c6427337e4da2f48ca12f413c9921f86f4f46488cb9a360"} Nov 03 20:40:03.300059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.300047 5043 scope.go:122] "RemoveContainer" containerID="cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec" Nov 03 20:40:03.300402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.300381 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj" Nov 03 20:40:03.316182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.316151 5043 scope.go:122] "RemoveContainer" containerID="cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec" Nov 03 20:40:03.316540 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:40:03.316504 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec\": container with ID starting with cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec not found: ID does not exist" containerID="cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec" Nov 03 20:40:03.316591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.316552 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec"} err="failed to get container status \"cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec\": rpc error: code = NotFound desc = could not find container \"cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec\": container with ID starting with cdc86525139022fcc80c90bd3ee02c5382352f49a3e85b301ca762e0ccbe84ec not found: ID does not exist" Nov 03 20:40:03.362834 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.362806 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj"] Nov 03 20:40:03.372814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:03.372788 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9fbxj"] Nov 03 20:40:04.676165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:04.676134 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" path="/var/lib/kubelet/pods/32b2a85c-e89f-4a83-a46e-48895b9ac3dc/volumes" Nov 03 20:40:05.627701 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:40:05.627659 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:40:32.766765 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:32.766673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:34.631346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:34.631295 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:34.740096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:34.740038 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:35.638979 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:40:35.638914 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:40:37.584656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:37.584165 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dl6vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:40:37.584656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:40:37.584231 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dl6vz" podUID="17893f17-0fe4-4378-b6cd-f2ee0dfebd26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:05.628135 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:41:05.628100 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:41:32.891220 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:41:32.891174 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:35.628531 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:41:35.628494 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:41:54.427257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:41:54.427160 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:05.631118 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:42:05.631060 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:42:33.069375 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:42:33.069311 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:33.169267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:42:33.169232 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:35.633319 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:42:35.633275 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:42:51.792839 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:42:51.792799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:05.631113 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:43:05.631055 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:43:21.975354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:21.975252 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:43:21.975354 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:21.975340 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:33.194501 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:33.194454 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:35.631055 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:43:35.631018 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:43:36.107450 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:36.107411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:36.207860 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:36.207820 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:39.256683 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:39.256653 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:43:39.258550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:39.258523 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:43:39.263925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:39.263903 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:43:39.265772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:43:39.265752 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:44:05.630425 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:44:05.630379 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:44:12.026348 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:12.026306 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:13.182303 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:13.182256 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:33.312952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:33.312918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:35.630034 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:44:35.629988 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:44:44.692086 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:44.692023 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:45.140049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:45.140002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:45.262363 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:45.262298 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:48.419596 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:48.419547 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:48.535897 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:48.535852 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:49.859923 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.859885 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94"] Nov 03 20:44:49.860463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860416 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerName="payload-processing" Nov 03 20:44:49.860463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860429 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerName="payload-processing" Nov 03 20:44:49.860463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860442 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a502b248-8079-481a-8837-f598a2f1829a" containerName="maas-api" Nov 03 20:44:49.860463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860447 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a502b248-8079-481a-8837-f598a2f1829a" containerName="maas-api" Nov 03 20:44:49.860463 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860463 5043 cpu_manager.go:414] "containerMap: removing container" podUID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" containerName="discovery" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860469 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" containerName="discovery" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860479 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a2e4fe77-f913-44d7-bc46-276176b1c65b" containerName="payload-processing" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860484 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e4fe77-f913-44d7-bc46-276176b1c65b" containerName="payload-processing" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860491 5043 cpu_manager.go:414] "containerMap: removing container" podUID="344ac58a-c539-4a11-85f5-1c285fd570bf" containerName="maas-api" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860496 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344ac58a-c539-4a11-85f5-1c285fd570bf" containerName="maas-api" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860502 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860506 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860516 5043 cpu_manager.go:414] "containerMap: removing container" podUID="51dfcaed-031e-4192-b2fb-e3a7a7cab90d" containerName="maas-api" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860520 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51dfcaed-031e-4192-b2fb-e3a7a7cab90d" containerName="maas-api" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860526 5043 cpu_manager.go:414] "containerMap: removing container" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" containerName="payload-pre-processing" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860531 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" containerName="payload-pre-processing" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860540 5043 cpu_manager.go:414] "containerMap: removing container" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860544 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860552 5043 cpu_manager.go:414] "containerMap: removing container" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860556 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860563 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860567 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860573 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860578 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860586 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860591 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860603 5043 cpu_manager.go:414] "containerMap: removing container" podUID="8971f511-2a1e-43d9-86bc-2b77098d9982" containerName="maas-api" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860609 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8971f511-2a1e-43d9-86bc-2b77098d9982" containerName="maas-api" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860615 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860620 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860632 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b49f54e4-98a5-46cc-b13c-8970fd0cc164" containerName="manager" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860637 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49f54e4-98a5-46cc-b13c-8970fd0cc164" containerName="manager" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860643 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" Nov 03 20:44:49.860635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860648 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860656 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0ba0abe0-719e-4208-a2d6-790a5c0dec00" containerName="payload-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860660 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba0abe0-719e-4208-a2d6-790a5c0dec00" containerName="payload-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860666 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5ea681b3-4632-45b8-80d4-333939e99c24" containerName="payload-pre-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860671 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea681b3-4632-45b8-80d4-333939e99c24" containerName="payload-pre-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860681 5043 cpu_manager.go:414] "containerMap: removing container" podUID="438055de-8964-4c06-a10b-786b96c44d8b" containerName="payload-pre-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860686 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438055de-8964-4c06-a10b-786b96c44d8b" containerName="payload-pre-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860692 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b5f72b17-0904-4497-8705-32c31f8b3b4d" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860696 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f72b17-0904-4497-8705-32c31f8b3b4d" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860702 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860706 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860713 5043 cpu_manager.go:414] "containerMap: removing container" podUID="95b70903-bf7e-469e-8542-78c3711b65ec" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860718 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b70903-bf7e-469e-8542-78c3711b65ec" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860787 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="438055de-8964-4c06-a10b-786b96c44d8b" containerName="payload-pre-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860797 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f72b17-0904-4497-8705-32c31f8b3b4d" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860802 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d835a0bf-ebcc-4919-9914-747f20a52fce" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860808 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860814 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b70903-bf7e-469e-8542-78c3711b65ec" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860821 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdb90f0-5685-43db-892b-d7f803178ad1" containerName="payload-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860825 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a502b248-8079-481a-8837-f598a2f1829a" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860831 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b2a85c-e89f-4a83-a46e-48895b9ac3dc" containerName="discovery" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860838 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e4fe77-f913-44d7-bc46-276176b1c65b" containerName="payload-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860844 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="344ac58a-c539-4a11-85f5-1c285fd570bf" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860850 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e535b50e-9c2f-470d-b676-5c9983c4ea65" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860856 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="51dfcaed-031e-4192-b2fb-e3a7a7cab90d" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860861 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="261cb760-7907-4d77-acf3-2f2cd927d303" containerName="payload-pre-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860866 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb39e4e7-e05b-4abd-b40f-24a79aeb7696" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860872 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="753bcdc7-60a4-4d92-9ca4-e11a33d4e339" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860878 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860882 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2859563-48c2-429d-ba23-b22f5b16b30e" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860887 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ffd880-d006-42cf-b4e5-93884993ca51" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860893 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8971f511-2a1e-43d9-86bc-2b77098d9982" containerName="maas-api" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860897 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae01033-875e-4913-9671-9a2464a837a6" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860903 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49f54e4-98a5-46cc-b13c-8970fd0cc164" containerName="manager" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860909 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff33112-7b0e-445f-a7c2-d8766dfa2d49" containerName="istio-proxy" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860915 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba0abe0-719e-4208-a2d6-790a5c0dec00" containerName="payload-processing" Nov 03 20:44:49.861547 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.860922 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea681b3-4632-45b8-80d4-333939e99c24" containerName="payload-pre-processing" Nov 03 20:44:49.864345 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.864325 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:49.882905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.882873 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-229f0a-serving-cert\"" Nov 03 20:44:49.899123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.899055 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94"] Nov 03 20:44:49.944166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.944123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:49.973893 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.973847 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:49.974123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.973992 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:49.974190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.974126 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:49.974249 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:49.974186 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5mr\" (UniqueName: \"kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.043787 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.043749 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:50.074881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.074832 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.075130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.074916 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5mr\" (UniqueName: \"kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.075130 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.075105 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.075261 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.075189 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.075901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.075873 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.077757 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.077725 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.077866 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.077817 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.100537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.100505 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5mr\" (UniqueName: \"kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr\") pod \"maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.146006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.145920 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:50.178317 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.178275 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:50.252635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.251301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:50.490383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.490350 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94"] Nov 03 20:44:50.494289 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:44:50.494249 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37a5a12_9df1_40e4_93fe_fa2f6e3c3a09.slice/crio-cd803f4faa833b36720c2df93aae16fbf352d723f35f3b1bd9423620c81b1752 WatchSource:0}: Error finding container cd803f4faa833b36720c2df93aae16fbf352d723f35f3b1bd9423620c81b1752: Status 404 returned error can't find the container with id cd803f4faa833b36720c2df93aae16fbf352d723f35f3b1bd9423620c81b1752 Nov 03 20:44:50.495885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:50.495858 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:44:51.273297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:51.273195 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" event={"ID":"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09","Type":"ContainerStarted","Data":"73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1"} Nov 03 20:44:51.273297 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:51.273268 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" event={"ID":"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09","Type":"ContainerStarted","Data":"cd803f4faa833b36720c2df93aae16fbf352d723f35f3b1bd9423620c81b1752"} Nov 03 20:44:51.273835 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:51.273399 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:51.313571 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:51.313509 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" podStartSLOduration=1.950131853 podStartE2EDuration="2.313491229s" totalImagesPullingTime="363.359376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:51.310976273 +0000 UTC m=+2472.777931526" watchObservedRunningTime="2026-11-03 20:44:51.313491229 +0000 UTC m=+2472.780446482" Nov 03 20:44:51.734177 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:51.734134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:53.463650 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:53.463609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:55.174275 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:55.174230 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:57.013647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:57.013612 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:57.286186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:57.286067 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:44:58.909810 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:44:58.909770 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:00.815029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:00.814989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:02.494339 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:02.494296 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:03.279415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:03.279375 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:03.893320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:03.893278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:04.034405 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:04.034369 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:04.221994 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:04.221885 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:04.358225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:04.358187 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:05.630649 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:45:05.630607 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:45:06.120475 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:06.120424 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:06.280935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:06.280890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:07.897122 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:07.897053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:09.612188 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:09.612135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:11.269891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:11.269850 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:12.872927 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:12.872874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:14.467755 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:14.467719 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:16.072056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:16.072015 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:17.773215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:17.773171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:19.065182 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:19.065134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:19.476084 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:19.476035 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:21.077936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:21.077903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:22.808869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:22.808829 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:24.760480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:24.760438 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:26.475292 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:26.475257 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:28.176493 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:28.176446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:29.778298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:29.778260 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:31.377663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:31.377627 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:33.019379 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:33.019344 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:33.415961 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:33.415925 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:34.682428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:34.682375 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:35.629389 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:45:35.629344 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:45:36.359139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:36.359095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:37.968386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:37.968345 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:39.719000 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:39.718959 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:41.471871 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:41.471833 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:43.158212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:43.158176 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:44.768772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:44.768729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:46.380082 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:46.380017 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:47.967546 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:47.967505 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:48.305143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:48.305024 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:49.716671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:49.716619 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:51.470052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:51.470010 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:53.076941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:53.076894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:53.361143 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:53.361036 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:54.776652 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:54.776599 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:56.394056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:56.394012 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:58.076476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:58.076438 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:59.090133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:59.090043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:59.190781 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:59.190735 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:59.790517 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:45:59.790477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:01.189783 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.189748 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94"] Nov 03 20:46:01.190246 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.190001 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" podUID="c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" containerName="maas-api" containerID="cri-o://73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1" gracePeriod=30 Nov 03 20:46:01.236495 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.236461 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:01.491267 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.491233 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:46:01.587231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.587196 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls\") pod \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " Nov 03 20:46:01.587231 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.587237 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs\") pod \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " Nov 03 20:46:01.587526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.587298 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr\" (UniqueName: \"kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr\") pod \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " Nov 03 20:46:01.587526 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.587388 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca\") pod \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\" (UID: \"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09\") " Nov 03 20:46:01.587878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.587844 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca" pod "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" (UID: "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:01.589606 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.589568 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr" pod "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" (UID: "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09"). InnerVolumeSpecName "kube-api-access-ng5mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:01.589713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.589644 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls" pod "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" (UID: "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:01.589713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.589679 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs" pod "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" (UID: "c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:01.688583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.688549 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:46:01.688583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.688578 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:46:01.688583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.688588 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng5mr\" (UniqueName: \"kubernetes.io/projected/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-kube-api-access-ng5mr\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:46:01.688844 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.688597 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:46:01.692364 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.692326 5043 generic.go:356] "Generic (PLEG): container finished" podID="c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" containerID="73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1" exitCode=0 Nov 03 20:46:01.692539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.692373 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" Nov 03 20:46:01.692539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.692387 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" event={"ID":"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09","Type":"ContainerDied","Data":"73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1"} Nov 03 20:46:01.692539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.692430 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94" event={"ID":"c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09","Type":"ContainerDied","Data":"cd803f4faa833b36720c2df93aae16fbf352d723f35f3b1bd9423620c81b1752"} Nov 03 20:46:01.692539 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.692452 5043 scope.go:122] "RemoveContainer" containerID="73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1" Nov 03 20:46:01.703052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.703027 5043 scope.go:122] "RemoveContainer" containerID="73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1" Nov 03 20:46:01.703453 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:46:01.703430 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1\": container with ID starting with 73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1 not found: ID does not exist" containerID="73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1" Nov 03 20:46:01.703508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.703464 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1"} err="failed to get container status \"73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1\": rpc error: code = NotFound desc = could not find container \"73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1\": container with ID starting with 73bc8fd1f79ffcb556fe6ccdf730200f4612fe5db9c16b1a0bea9867415d3df1 not found: ID does not exist" Nov 03 20:46:01.743356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.743264 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94"] Nov 03 20:46:01.747842 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.747808 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-229f0a-547666f5cd-jrb94"] Nov 03 20:46:01.766732 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.766689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:01.867360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:01.867316 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:02.365345 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:02.365293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:02.677431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:02.677347 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" path="/var/lib/kubelet/pods/c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09/volumes" Nov 03 20:46:05.630205 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:46:05.630168 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:46:08.052623 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:08.052579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:08.144562 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:08.144518 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:08.368979 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:08.368936 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:30.113201 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:30.113102 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:30.702322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:30.702282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:33.532756 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.532718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:33.647125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.647085 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd"] Nov 03 20:46:33.647335 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.647312 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" podUID="5472f63c-7e21-4301-ad70-57a7c9760b82" containerName="manager" containerID="cri-o://f3129ebecd0fae0e647475c6bfc9b1ee4c3ad39991a29678a4c25cb4a442e23f" gracePeriod=10 Nov 03 20:46:33.914635 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.914603 5043 generic.go:356] "Generic (PLEG): container finished" podID="5472f63c-7e21-4301-ad70-57a7c9760b82" containerID="f3129ebecd0fae0e647475c6bfc9b1ee4c3ad39991a29678a4c25cb4a442e23f" exitCode=0 Nov 03 20:46:33.914791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.914679 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" event={"ID":"5472f63c-7e21-4301-ad70-57a7c9760b82","Type":"ContainerDied","Data":"f3129ebecd0fae0e647475c6bfc9b1ee4c3ad39991a29678a4c25cb4a442e23f"} Nov 03 20:46:33.914791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.914734 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" event={"ID":"5472f63c-7e21-4301-ad70-57a7c9760b82","Type":"ContainerDied","Data":"0210aaa9ef14ce9c4c93eaad2f8caa095201c2eab5227b151a04a2376bc8354f"} Nov 03 20:46:33.914791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.914752 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0210aaa9ef14ce9c4c93eaad2f8caa095201c2eab5227b151a04a2376bc8354f" Nov 03 20:46:33.919503 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.919479 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:46:33.969334 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.969287 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume\") pod \"5472f63c-7e21-4301-ad70-57a7c9760b82\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " Nov 03 20:46:33.969536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.969381 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w\" (UniqueName: \"kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w\") pod \"5472f63c-7e21-4301-ad70-57a7c9760b82\" (UID: \"5472f63c-7e21-4301-ad70-57a7c9760b82\") " Nov 03 20:46:33.969695 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.969669 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume" pod "5472f63c-7e21-4301-ad70-57a7c9760b82" (UID: "5472f63c-7e21-4301-ad70-57a7c9760b82"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:33.969832 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.969810 5043 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5472f63c-7e21-4301-ad70-57a7c9760b82-extensions-socket-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:46:33.976649 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:33.976616 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w" pod "5472f63c-7e21-4301-ad70-57a7c9760b82" (UID: "5472f63c-7e21-4301-ad70-57a7c9760b82"). InnerVolumeSpecName "kube-api-access-bgh9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:34.071313 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:34.071222 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgh9w\" (UniqueName: \"kubernetes.io/projected/5472f63c-7e21-4301-ad70-57a7c9760b82-kube-api-access-bgh9w\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:46:34.920462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:34.920365 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd" Nov 03 20:46:34.970374 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:34.970335 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd"] Nov 03 20:46:34.982541 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:34.982501 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lm2pd"] Nov 03 20:46:35.629446 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:46:35.629408 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:46:36.172349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:36.172315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:36.676933 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:36.676894 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5472f63c-7e21-4301-ad70-57a7c9760b82" path="/var/lib/kubelet/pods/5472f63c-7e21-4301-ad70-57a7c9760b82/volumes" Nov 03 20:46:39.835132 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:46:39.835099 5043 scope.go:122] "RemoveContainer" containerID="f3129ebecd0fae0e647475c6bfc9b1ee4c3ad39991a29678a4c25cb4a442e23f" Nov 03 20:47:05.629228 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:47:05.629188 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:47:25.458920 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:25.458873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:33.658622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:33.658576 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:35.629260 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:47:35.629213 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:47:36.725794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.725753 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm"] Nov 03 20:47:36.726328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726241 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5472f63c-7e21-4301-ad70-57a7c9760b82" containerName="manager" Nov 03 20:47:36.726328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726253 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5472f63c-7e21-4301-ad70-57a7c9760b82" containerName="manager" Nov 03 20:47:36.726328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726271 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" Nov 03 20:47:36.726328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726276 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be65ccf6-55cd-426f-8c1a-b677eddeb02f" containerName="istio-proxy" Nov 03 20:47:36.726328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726302 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" containerName="maas-api" Nov 03 20:47:36.726328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726307 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" containerName="maas-api" Nov 03 20:47:36.726527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726374 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37a5a12-9df1-40e4-93fe-fa2f6e3c3a09" containerName="maas-api" Nov 03 20:47:36.726527 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.726386 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5472f63c-7e21-4301-ad70-57a7c9760b82" containerName="manager" Nov 03 20:47:36.729721 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.729698 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:36.744112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.744064 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm"] Nov 03 20:47:36.752970 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.752896 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fz5q8\"" Nov 03 20:47:36.809969 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.809929 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpkj\" (UniqueName: \"kubernetes.io/projected/715fba24-96d9-4967-a526-25859e8ea8e1-kube-api-access-kcpkj\") pod \"kuadrant-operator-controller-manager-678d87766f-4jslm\" (UID: \"715fba24-96d9-4967-a526-25859e8ea8e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:36.810193 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.809980 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/715fba24-96d9-4967-a526-25859e8ea8e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4jslm\" (UID: \"715fba24-96d9-4967-a526-25859e8ea8e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:36.911056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.911012 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/715fba24-96d9-4967-a526-25859e8ea8e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4jslm\" (UID: \"715fba24-96d9-4967-a526-25859e8ea8e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:36.911280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.911179 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpkj\" (UniqueName: \"kubernetes.io/projected/715fba24-96d9-4967-a526-25859e8ea8e1-kube-api-access-kcpkj\") pod \"kuadrant-operator-controller-manager-678d87766f-4jslm\" (UID: \"715fba24-96d9-4967-a526-25859e8ea8e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:36.911482 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.911457 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/715fba24-96d9-4967-a526-25859e8ea8e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4jslm\" (UID: \"715fba24-96d9-4967-a526-25859e8ea8e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:36.936747 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:36.936715 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpkj\" (UniqueName: \"kubernetes.io/projected/715fba24-96d9-4967-a526-25859e8ea8e1-kube-api-access-kcpkj\") pod \"kuadrant-operator-controller-manager-678d87766f-4jslm\" (UID: \"715fba24-96d9-4967-a526-25859e8ea8e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:37.044041 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:37.043948 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:37.387211 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:37.387180 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm"] Nov 03 20:47:37.389969 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:47:37.389939 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715fba24_96d9_4967_a526_25859e8ea8e1.slice/crio-d98c72847b9e013633210daaa737d9856871ae3beaf669ab10da1c0d10c52738 WatchSource:0}: Error finding container d98c72847b9e013633210daaa737d9856871ae3beaf669ab10da1c0d10c52738: Status 404 returned error can't find the container with id d98c72847b9e013633210daaa737d9856871ae3beaf669ab10da1c0d10c52738 Nov 03 20:47:38.271502 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:38.271461 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" event={"ID":"715fba24-96d9-4967-a526-25859e8ea8e1","Type":"ContainerStarted","Data":"f809d97be8ac78095979a08649e998db95c091368e9415814e81a70bf2f08efd"} Nov 03 20:47:38.271913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:38.271531 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" event={"ID":"715fba24-96d9-4967-a526-25859e8ea8e1","Type":"ContainerStarted","Data":"d98c72847b9e013633210daaa737d9856871ae3beaf669ab10da1c0d10c52738"} Nov 03 20:47:38.271913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:38.271553 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:38.339661 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:38.339598 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" podStartSLOduration=2.3395833169999998 podStartE2EDuration="2.339583317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:38.326432441 +0000 UTC m=+2639.793387695" watchObservedRunningTime="2026-11-03 20:47:38.339583317 +0000 UTC m=+2639.806538570" Nov 03 20:47:49.279701 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:49.279666 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4jslm" Nov 03 20:47:57.104807 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:57.104718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:57.206239 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:47:57.206199 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:05.628182 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:48:05.628141 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:48:17.559033 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:17.558972 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:33.776639 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:33.776601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:34.161792 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:34.161747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:34.746114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:34.746011 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:34.856874 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:34.856835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:35.629476 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:48:35.629427 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:48:35.655428 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:35.655400 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:35.774739 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:35.774707 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.119411 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:38.119373 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.226622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:38.226591 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:39.299291 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.299259 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:48:39.302244 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.302219 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:48:39.306532 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.306516 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:48:39.309696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.309679 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:48:39.583778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.583743 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h"] Nov 03 20:48:39.596446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.596386 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.610109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.610049 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h"] Nov 03 20:48:39.618001 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.617968 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0d89d1-serving-cert\"" Nov 03 20:48:39.664668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.664567 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:39.697690 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.697656 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.697873 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.697701 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.697942 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.697905 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.697998 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.697941 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgbvh\" (UniqueName: \"kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.775052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.775013 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:39.799355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.799322 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.799355 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.799357 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbvh\" (UniqueName: \"kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.799621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.799406 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.799621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.799432 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.800178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.800146 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.801878 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.801849 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.802123 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.802099 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.818054 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.818026 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbvh\" (UniqueName: \"kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh\") pod \"maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.872589 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.872501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:39.918914 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.918878 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:39.971556 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:39.971506 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:40.228338 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:40.228311 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h"] Nov 03 20:48:40.230367 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:48:40.230337 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca066045_0052_49b7_8c00_951d9b9b1009.slice/crio-4fbd6627a3f271edc83737d6bd68b6fb0132f1b41b0a323e18684e3d80786e39 WatchSource:0}: Error finding container 4fbd6627a3f271edc83737d6bd68b6fb0132f1b41b0a323e18684e3d80786e39: Status 404 returned error can't find the container with id 4fbd6627a3f271edc83737d6bd68b6fb0132f1b41b0a323e18684e3d80786e39 Nov 03 20:48:40.647980 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:40.647947 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" event={"ID":"ca066045-0052-49b7-8c00-951d9b9b1009","Type":"ContainerStarted","Data":"4fbd6627a3f271edc83737d6bd68b6fb0132f1b41b0a323e18684e3d80786e39"} Nov 03 20:48:41.420958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:41.420906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:41.656909 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:41.656870 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" event={"ID":"ca066045-0052-49b7-8c00-951d9b9b1009","Type":"ContainerStarted","Data":"7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7"} Nov 03 20:48:41.657402 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:41.656941 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:41.698257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:41.698144 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" podStartSLOduration=2.139202125 podStartE2EDuration="2.6981272s" totalImagesPullingTime="558.925075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:41.697470805 +0000 UTC m=+2703.164426078" watchObservedRunningTime="2026-11-03 20:48:41.6981272 +0000 UTC m=+2703.165082454" Nov 03 20:48:43.066333 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:43.066285 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:44.800371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:44.800321 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:46.495838 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:46.495790 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:47.668015 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:47.667982 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:48:48.093697 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:48.093663 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:49.814858 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:49.814819 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:51.613422 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:51.613385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:51.712185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:51.712134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:52.925299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:52.925263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.490501 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:53.490457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.597812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:53.597777 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.709222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:53.709155 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:55.298560 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:55.298518 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:55.507481 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:55.507446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:57.109905 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:57.109858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:58.918593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:48:58.918559 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:00.673409 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:00.673373 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:02.373440 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:02.373401 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:03.976962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:03.976924 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:05.628983 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:49:05.628951 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:49:05.676167 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:05.676134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:07.469272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:07.469234 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:09.181577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:09.181529 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:10.793570 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:10.793535 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:12.600461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:12.600422 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:14.372318 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:14.372283 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:15.958045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:15.958007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:17.689965 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:17.689927 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:19.417395 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:19.417354 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:21.078459 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:21.078356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:22.340756 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:22.340716 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:22.454119 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:22.454061 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:22.680059 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:22.679965 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:24.372824 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:24.372790 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:26.073386 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:26.073348 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:27.780012 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:27.779975 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:29.464328 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:29.464291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:31.179424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:31.179387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:32.763864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:32.763810 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:33.884618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:33.884582 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:34.361712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:34.361672 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:35.628059 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:49:35.628028 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:49:35.962849 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:35.962770 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:37.561400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:37.561307 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:38.073164 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:38.073125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:39.397371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:39.397327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:41.086415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:41.086376 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:42.658581 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:42.658545 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:43.095750 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:43.095708 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:44.399854 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:44.399818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:46.071797 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:46.071760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:47.175298 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:47.175267 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:47.275451 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:47.275419 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:47.779403 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:47.779357 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:48.880774 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:48.880729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:49.001282 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:49.001244 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:49.615703 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:49.615665 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:50.925250 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:50.925213 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h"] Nov 03 20:49:50.925693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:50.925437 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" podUID="ca066045-0052-49b7-8c00-951d9b9b1009" containerName="maas-api" containerID="cri-o://7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7" gracePeriod=30 Nov 03 20:49:50.982414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:50.982372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:51.404353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.404319 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:49:51.491414 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.491320 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:51.552223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.552185 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs\") pod \"ca066045-0052-49b7-8c00-951d9b9b1009\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " Nov 03 20:49:51.552435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.552308 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls\") pod \"ca066045-0052-49b7-8c00-951d9b9b1009\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " Nov 03 20:49:51.552435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.552373 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh\" (UniqueName: \"kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh\") pod \"ca066045-0052-49b7-8c00-951d9b9b1009\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " Nov 03 20:49:51.552435 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.552394 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca\") pod \"ca066045-0052-49b7-8c00-951d9b9b1009\" (UID: \"ca066045-0052-49b7-8c00-951d9b9b1009\") " Nov 03 20:49:51.552820 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.552794 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca" pod "ca066045-0052-49b7-8c00-951d9b9b1009" (UID: "ca066045-0052-49b7-8c00-951d9b9b1009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:51.554391 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.554366 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs" pod "ca066045-0052-49b7-8c00-951d9b9b1009" (UID: "ca066045-0052-49b7-8c00-951d9b9b1009"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:51.554480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.554392 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh" pod "ca066045-0052-49b7-8c00-951d9b9b1009" (UID: "ca066045-0052-49b7-8c00-951d9b9b1009"). InnerVolumeSpecName "kube-api-access-fgbvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:51.554480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.554415 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls" pod "ca066045-0052-49b7-8c00-951d9b9b1009" (UID: "ca066045-0052-49b7-8c00-951d9b9b1009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:51.596026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.595986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:51.653310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.653273 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:49:51.653310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.653303 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgbvh\" (UniqueName: \"kubernetes.io/projected/ca066045-0052-49b7-8c00-951d9b9b1009-kube-api-access-fgbvh\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:49:51.653310 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.653315 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca066045-0052-49b7-8c00-951d9b9b1009-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:49:51.653567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:51.653325 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca066045-0052-49b7-8c00-951d9b9b1009-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:49:52.069627 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.069587 5043 generic.go:356] "Generic (PLEG): container finished" podID="ca066045-0052-49b7-8c00-951d9b9b1009" containerID="7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7" exitCode=0 Nov 03 20:49:52.070094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.069642 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" Nov 03 20:49:52.070094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.069674 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" event={"ID":"ca066045-0052-49b7-8c00-951d9b9b1009","Type":"ContainerDied","Data":"7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7"} Nov 03 20:49:52.070094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.069747 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h" event={"ID":"ca066045-0052-49b7-8c00-951d9b9b1009","Type":"ContainerDied","Data":"4fbd6627a3f271edc83737d6bd68b6fb0132f1b41b0a323e18684e3d80786e39"} Nov 03 20:49:52.070094 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.069777 5043 scope.go:122] "RemoveContainer" containerID="7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7" Nov 03 20:49:52.081037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.081013 5043 scope.go:122] "RemoveContainer" containerID="7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7" Nov 03 20:49:52.081367 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:49:52.081339 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7\": container with ID starting with 7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7 not found: ID does not exist" containerID="7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7" Nov 03 20:49:52.081473 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.081369 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7"} err="failed to get container status \"7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7\": rpc error: code = NotFound desc = could not find container \"7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7\": container with ID starting with 7507c2f9b0b20261ad6c78079dbccc637979adef36f2a975a466210065b4b0c7 not found: ID does not exist" Nov 03 20:49:52.128233 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.128196 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h"] Nov 03 20:49:52.139446 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.139245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:52.141314 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.141287 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d89d1-dd9b5dddb-w8z7h"] Nov 03 20:49:52.676006 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:52.675971 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca066045-0052-49b7-8c00-951d9b9b1009" path="/var/lib/kubelet/pods/ca066045-0052-49b7-8c00-951d9b9b1009/volumes" Nov 03 20:49:57.822645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:57.822600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:57.936833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:57.936790 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:58.157410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:49:58.157373 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:01.469876 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:01.469836 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:01.565079 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:01.565015 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:02.347301 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:02.347266 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:05.731007 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:50:05.730960 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:50:06.967553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:06.967244 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5mbh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:50:06.967553 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:06.967350 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5mbh8" podUID="2044f915-8f34-4c29-ac6b-754910c41df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:19.022200 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.022165 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22"] Nov 03 20:50:19.022696 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.022681 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ca066045-0052-49b7-8c00-951d9b9b1009" containerName="maas-api" Nov 03 20:50:19.022743 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.022699 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca066045-0052-49b7-8c00-951d9b9b1009" containerName="maas-api" Nov 03 20:50:19.022798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.022789 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca066045-0052-49b7-8c00-951d9b9b1009" containerName="maas-api" Nov 03 20:50:19.026103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.026080 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.045323 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.045294 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22"] Nov 03 20:50:19.058862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.058829 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1b6f264d-kserve-self-signed-certs\"" Nov 03 20:50:19.075642 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.075611 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:19.185934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.185895 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.185934 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.185935 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.186230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.185959 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjl9\" (UniqueName: \"kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.186230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.186097 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.186230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.186212 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.186360 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.186333 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287126 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287026 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287136 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287163 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287182 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjl9\" (UniqueName: \"kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287203 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287299 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287268 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287593 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287720 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287646 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.287785 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.287723 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.289671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.289650 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.289779 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.289697 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.306199 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.306175 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjl9\" (UniqueName: \"kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9\") pod \"e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.350845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.350813 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:19.653881 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.653846 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22"] Nov 03 20:50:19.657193 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:50:19.657168 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a5ec111_a421_447e_af54_93f0a92ab203.slice/crio-b343a504eef7c6128e83351703850214a3e322322b9ea09d1c23815492987789 WatchSource:0}: Error finding container b343a504eef7c6128e83351703850214a3e322322b9ea09d1c23815492987789: Status 404 returned error can't find the container with id b343a504eef7c6128e83351703850214a3e322322b9ea09d1c23815492987789 Nov 03 20:50:19.981434 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:19.981340 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:20.246257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:20.246155 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" event={"ID":"4a5ec111-a421-447e-af54-93f0a92ab203","Type":"ContainerStarted","Data":"5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba"} Nov 03 20:50:20.246257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:20.246215 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" event={"ID":"4a5ec111-a421-447e-af54-93f0a92ab203","Type":"ContainerStarted","Data":"b343a504eef7c6128e83351703850214a3e322322b9ea09d1c23815492987789"} Nov 03 20:50:20.283372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:20.283318 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" podStartSLOduration=2.283302985 podStartE2EDuration="2.283302985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:20.282858164 +0000 UTC m=+2801.749813419" watchObservedRunningTime="2026-11-03 20:50:20.283302985 +0000 UTC m=+2801.750258238" Nov 03 20:50:20.351512 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:20.351477 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:20.386052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:20.386014 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:29.351610 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:29.351577 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:29.364154 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:29.364127 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:30.322895 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:30.322864 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:50:34.002222 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:34.002172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:34.668155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:34.668120 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:35.629225 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:50:35.629190 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:50:51.808840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:51.808754 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:53.717666 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:53.717622 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22"] Nov 03 20:50:53.752467 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:50:53.752422 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:05.629333 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:51:05.629294 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:51:08.740580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:08.740516 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" podUID="4a5ec111-a421-447e-af54-93f0a92ab203" containerName="main" containerID="cri-o://5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba" gracePeriod=45 Nov 03 20:51:08.991038 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:08.990965 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:51:09.055404 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055306 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9\" (UniqueName: \"kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9\") pod \"4a5ec111-a421-447e-af54-93f0a92ab203\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " Nov 03 20:51:09.055599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055435 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home\") pod \"4a5ec111-a421-447e-af54-93f0a92ab203\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " Nov 03 20:51:09.055599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055476 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache\") pod \"4a5ec111-a421-447e-af54-93f0a92ab203\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " Nov 03 20:51:09.055599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055538 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir\") pod \"4a5ec111-a421-447e-af54-93f0a92ab203\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " Nov 03 20:51:09.055599 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055573 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm\") pod \"4a5ec111-a421-447e-af54-93f0a92ab203\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " Nov 03 20:51:09.055840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055667 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs\") pod \"4a5ec111-a421-447e-af54-93f0a92ab203\" (UID: \"4a5ec111-a421-447e-af54-93f0a92ab203\") " Nov 03 20:51:09.055840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055721 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home" pod "4a5ec111-a421-447e-af54-93f0a92ab203" (UID: "4a5ec111-a421-447e-af54-93f0a92ab203"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:09.055840 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055818 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache" pod "4a5ec111-a421-447e-af54-93f0a92ab203" (UID: "4a5ec111-a421-447e-af54-93f0a92ab203"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:09.056002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.055933 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir" pod "4a5ec111-a421-447e-af54-93f0a92ab203" (UID: "4a5ec111-a421-447e-af54-93f0a92ab203"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:09.056194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.056166 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-tmp-dir\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:09.056194 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.056195 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-home\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:09.056429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.056210 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-model-cache\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:09.058443 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.058418 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm" pod "4a5ec111-a421-447e-af54-93f0a92ab203" (UID: "4a5ec111-a421-447e-af54-93f0a92ab203"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:09.058597 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.058573 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs" pod "4a5ec111-a421-447e-af54-93f0a92ab203" (UID: "4a5ec111-a421-447e-af54-93f0a92ab203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:09.058775 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.058751 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9" pod "4a5ec111-a421-447e-af54-93f0a92ab203" (UID: "4a5ec111-a421-447e-af54-93f0a92ab203"). InnerVolumeSpecName "kube-api-access-drjl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:09.157973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.157882 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5ec111-a421-447e-af54-93f0a92ab203-dshm\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:09.157973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.157926 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5ec111-a421-447e-af54-93f0a92ab203-tls-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:09.157973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.157944 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjl9\" (UniqueName: \"kubernetes.io/projected/4a5ec111-a421-447e-af54-93f0a92ab203-kube-api-access-drjl9\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:09.660215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.641306 5043 generic.go:356] "Generic (PLEG): container finished" podID="4a5ec111-a421-447e-af54-93f0a92ab203" containerID="5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba" exitCode=0 Nov 03 20:51:09.660215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.641522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" event={"ID":"4a5ec111-a421-447e-af54-93f0a92ab203","Type":"ContainerDied","Data":"5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba"} Nov 03 20:51:09.660215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.641567 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" event={"ID":"4a5ec111-a421-447e-af54-93f0a92ab203","Type":"ContainerDied","Data":"b343a504eef7c6128e83351703850214a3e322322b9ea09d1c23815492987789"} Nov 03 20:51:09.660215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.641595 5043 scope.go:122] "RemoveContainer" containerID="5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba" Nov 03 20:51:09.660215 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.641766 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22" Nov 03 20:51:09.672805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.672778 5043 scope.go:122] "RemoveContainer" containerID="5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba" Nov 03 20:51:09.673114 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:51:09.673064 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba\": container with ID starting with 5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba not found: ID does not exist" containerID="5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba" Nov 03 20:51:09.673210 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.673125 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba"} err="failed to get container status \"5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba\": rpc error: code = NotFound desc = could not find container \"5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba\": container with ID starting with 5323259fd91a4069b4009a5a5d7a9642d3aa6c6a9da27f94ecb365aae44388ba not found: ID does not exist" Nov 03 20:51:09.720290 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.720220 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22"] Nov 03 20:51:09.735625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:09.735591 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1b6f264d-kserve-54c6cb6f5c-rxv22"] Nov 03 20:51:10.696097 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:10.678912 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5ec111-a421-447e-af54-93f0a92ab203" path="/var/lib/kubelet/pods/4a5ec111-a421-447e-af54-93f0a92ab203/volumes" Nov 03 20:51:32.128773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.128741 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h"] Nov 03 20:51:32.129425 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.129406 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4a5ec111-a421-447e-af54-93f0a92ab203" containerName="main" Nov 03 20:51:32.129472 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.129427 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5ec111-a421-447e-af54-93f0a92ab203" containerName="main" Nov 03 20:51:32.129523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.129513 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5ec111-a421-447e-af54-93f0a92ab203" containerName="main" Nov 03 20:51:32.134664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.134641 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.163632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.163600 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h"] Nov 03 20:51:32.225529 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.225497 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67g5\" (UniqueName: \"kubernetes.io/projected/344c8e8e-1434-4e8e-be26-bed9f902dca1-kube-api-access-j67g5\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.225722 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.225631 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344c8e8e-1434-4e8e-be26-bed9f902dca1-metrics-certs\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.225722 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.225657 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344c8e8e-1434-4e8e-be26-bed9f902dca1-openshift-service-ca\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.225813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.225764 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344c8e8e-1434-4e8e-be26-bed9f902dca1-maas-api-tls\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.326535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.326503 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67g5\" (UniqueName: \"kubernetes.io/projected/344c8e8e-1434-4e8e-be26-bed9f902dca1-kube-api-access-j67g5\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.326712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.326571 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344c8e8e-1434-4e8e-be26-bed9f902dca1-metrics-certs\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.326712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.326591 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344c8e8e-1434-4e8e-be26-bed9f902dca1-openshift-service-ca\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.326712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.326613 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344c8e8e-1434-4e8e-be26-bed9f902dca1-maas-api-tls\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.327369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.327340 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/344c8e8e-1434-4e8e-be26-bed9f902dca1-openshift-service-ca\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.329032 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.329009 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/344c8e8e-1434-4e8e-be26-bed9f902dca1-maas-api-tls\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.329176 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.329035 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344c8e8e-1434-4e8e-be26-bed9f902dca1-metrics-certs\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.353087 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.353047 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67g5\" (UniqueName: \"kubernetes.io/projected/344c8e8e-1434-4e8e-be26-bed9f902dca1-kube-api-access-j67g5\") pod \"maas-api-796c56b65d-pzs9h\" (UID: \"344c8e8e-1434-4e8e-be26-bed9f902dca1\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.452891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.452822 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:32.754982 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.754747 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h"] Nov 03 20:51:32.758168 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:51:32.758130 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344c8e8e_1434_4e8e_be26_bed9f902dca1.slice/crio-0f71b80259fe3c3ce2a9d5dc1d7765b530518a891855471fcb13e42d731ddb38 WatchSource:0}: Error finding container 0f71b80259fe3c3ce2a9d5dc1d7765b530518a891855471fcb13e42d731ddb38: Status 404 returned error can't find the container with id 0f71b80259fe3c3ce2a9d5dc1d7765b530518a891855471fcb13e42d731ddb38 Nov 03 20:51:32.759609 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.759587 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:51:32.808712 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:32.808686 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" event={"ID":"344c8e8e-1434-4e8e-be26-bed9f902dca1","Type":"ContainerStarted","Data":"0f71b80259fe3c3ce2a9d5dc1d7765b530518a891855471fcb13e42d731ddb38"} Nov 03 20:51:33.816157 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:33.816110 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" event={"ID":"344c8e8e-1434-4e8e-be26-bed9f902dca1","Type":"ContainerStarted","Data":"dc2743588891adee0c83a351e89ed3ea201b33411e87702e5f0e068721142616"} Nov 03 20:51:33.816659 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:33.816190 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:33.856166 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:33.856060 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" podStartSLOduration=1.449707215 podStartE2EDuration="1.856046872s" totalImagesPullingTime="406.339657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:33.856011805 +0000 UTC m=+2875.322967072" watchObservedRunningTime="2026-11-03 20:51:33.856046872 +0000 UTC m=+2875.323002125" Nov 03 20:51:34.127256 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:34.127218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:34.245092 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:34.245042 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:35.628969 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:51:35.628933 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:51:38.814268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:38.814230 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:39.263216 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:39.263180 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:39.376645 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:39.376607 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:39.827424 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:39.827398 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796c56b65d-pzs9h" Nov 03 20:51:39.898534 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:39.898504 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm"] Nov 03 20:51:39.898772 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:39.898736 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" podUID="319c0201-cbdc-49d2-bfec-d88a7e416e16" containerName="maas-api" containerID="cri-o://17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531" gracePeriod=30 Nov 03 20:51:41.175347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.175322 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:51:41.321664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.321589 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls\") pod \"319c0201-cbdc-49d2-bfec-d88a7e416e16\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " Nov 03 20:51:41.321664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.321633 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs\") pod \"319c0201-cbdc-49d2-bfec-d88a7e416e16\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " Nov 03 20:51:41.321664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.321660 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca\") pod \"319c0201-cbdc-49d2-bfec-d88a7e416e16\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " Nov 03 20:51:41.321891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.321807 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85\" (UniqueName: \"kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85\") pod \"319c0201-cbdc-49d2-bfec-d88a7e416e16\" (UID: \"319c0201-cbdc-49d2-bfec-d88a7e416e16\") " Nov 03 20:51:41.322114 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.322058 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca" pod "319c0201-cbdc-49d2-bfec-d88a7e416e16" (UID: "319c0201-cbdc-49d2-bfec-d88a7e416e16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:41.322255 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.322238 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/319c0201-cbdc-49d2-bfec-d88a7e416e16-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:41.323896 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.323871 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85" pod "319c0201-cbdc-49d2-bfec-d88a7e416e16" (UID: "319c0201-cbdc-49d2-bfec-d88a7e416e16"). InnerVolumeSpecName "kube-api-access-mfp85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:41.324005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.323907 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls" pod "319c0201-cbdc-49d2-bfec-d88a7e416e16" (UID: "319c0201-cbdc-49d2-bfec-d88a7e416e16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:41.324005 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.323939 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs" pod "319c0201-cbdc-49d2-bfec-d88a7e416e16" (UID: "319c0201-cbdc-49d2-bfec-d88a7e416e16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:41.423144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.423111 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfp85\" (UniqueName: \"kubernetes.io/projected/319c0201-cbdc-49d2-bfec-d88a7e416e16-kube-api-access-mfp85\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:41.423144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.423144 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:41.423144 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.423159 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/319c0201-cbdc-49d2-bfec-d88a7e416e16-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:51:41.868091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.868045 5043 generic.go:356] "Generic (PLEG): container finished" podID="319c0201-cbdc-49d2-bfec-d88a7e416e16" containerID="17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531" exitCode=0 Nov 03 20:51:41.868271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.868112 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" Nov 03 20:51:41.868271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.868128 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" event={"ID":"319c0201-cbdc-49d2-bfec-d88a7e416e16","Type":"ContainerDied","Data":"17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531"} Nov 03 20:51:41.868271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.868183 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm" event={"ID":"319c0201-cbdc-49d2-bfec-d88a7e416e16","Type":"ContainerDied","Data":"0423ecf0af37e911759d6286c2fc90c52a8b93658131e3c50f75e4652afacd30"} Nov 03 20:51:41.868271 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.868205 5043 scope.go:122] "RemoveContainer" containerID="17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531" Nov 03 20:51:41.880098 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.880063 5043 scope.go:122] "RemoveContainer" containerID="17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531" Nov 03 20:51:41.880378 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:51:41.880359 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531\": container with ID starting with 17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531 not found: ID does not exist" containerID="17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531" Nov 03 20:51:41.880462 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.880385 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531"} err="failed to get container status \"17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531\": rpc error: code = NotFound desc = could not find container \"17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531\": container with ID starting with 17b3954e23357b34da5873b1802220a2c37660f5c582adca72439412cfca7531 not found: ID does not exist" Nov 03 20:51:41.885133 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.884997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:41.933399 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.933366 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm"] Nov 03 20:51:41.945799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:41.945769 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-jg5bm"] Nov 03 20:51:42.012480 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:42.012447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:42.676742 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:42.676710 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319c0201-cbdc-49d2-bfec-d88a7e416e16" path="/var/lib/kubelet/pods/319c0201-cbdc-49d2-bfec-d88a7e416e16/volumes" Nov 03 20:51:43.358055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.358025 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn"] Nov 03 20:51:43.358499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.358484 5043 cpu_manager.go:414] "containerMap: removing container" podUID="319c0201-cbdc-49d2-bfec-d88a7e416e16" containerName="maas-api" Nov 03 20:51:43.358551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.358500 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319c0201-cbdc-49d2-bfec-d88a7e416e16" containerName="maas-api" Nov 03 20:51:43.358625 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.358617 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="319c0201-cbdc-49d2-bfec-d88a7e416e16" containerName="maas-api" Nov 03 20:51:43.362977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.362956 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.382551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.382528 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn"] Nov 03 20:51:43.391667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.391642 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e2966c90-serving-cert\"" Nov 03 20:51:43.474852 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.474809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:43.547109 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.547053 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.547499 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.547472 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.547706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.547688 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.547884 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.547870 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4hf\" (UniqueName: \"kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.578241 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.578199 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:43.649591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.649511 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.649591 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.649558 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.649793 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.649606 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4hf\" (UniqueName: \"kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.649793 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.649675 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.649793 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:51:43.649728 5043 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-serving-cert: secret "maas-api-e2e-ait-e2966c90-serving-cert" not found Nov 03 20:51:43.649956 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:51:43.649821 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls podName:2a9c378e-f855-4925-af28-a2abc3124493 nodeName:}" failed. No retries permitted until 2026-11-03 20:51:44.149801004 +0000 UTC m=+2885.616756241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls") pod "maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" (UID: "2a9c378e-f855-4925-af28-a2abc3124493") : secret "maas-api-e2e-ait-e2966c90-serving-cert" not found Nov 03 20:51:43.650353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.650334 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.652103 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.652059 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.670450 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.670423 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4hf\" (UniqueName: \"kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:43.869618 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:43.869581 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.153647 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.153614 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:44.156218 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.156193 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\") pod \"maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:44.261619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.261580 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.276608 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.276577 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:44.379259 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.379210 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.501762 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.501691 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm"] Nov 03 20:51:44.506476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.506456 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.523612 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.523583 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm"] Nov 03 20:51:44.543383 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.543356 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e2966c90\"" Nov 03 20:51:44.543508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.543398 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e2966c90-dockercfg-5szr6\"" Nov 03 20:51:44.609039 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.608331 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn"] Nov 03 20:51:44.611507 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:51:44.611420 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9c378e_f855_4925_af28_a2abc3124493.slice/crio-09c470852a2397e181eb47c79c4f315a2d48c44d9a42117b688270d50271d5a6 WatchSource:0}: Error finding container 09c470852a2397e181eb47c79c4f315a2d48c44d9a42117b688270d50271d5a6: Status 404 returned error can't find the container with id 09c470852a2397e181eb47c79c4f315a2d48c44d9a42117b688270d50271d5a6 Nov 03 20:51:44.658155 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.658123 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.658295 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.658178 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89dn\" (UniqueName: \"kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn\") pod \"payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.764805 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.764705 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.765519 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.765488 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89dn\" (UniqueName: \"kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn\") pod \"payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.765812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.765787 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.784621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.784597 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89dn\" (UniqueName: \"kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn\") pod \"payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.820748 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.820717 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:44.896745 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:44.896707 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" event={"ID":"2a9c378e-f855-4925-af28-a2abc3124493","Type":"ContainerStarted","Data":"09c470852a2397e181eb47c79c4f315a2d48c44d9a42117b688270d50271d5a6"} Nov 03 20:51:45.130112 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.130086 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm"] Nov 03 20:51:45.132388 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:51:45.132355 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4492c45a_e540_4512_9ee9_f137628c4ed3.slice/crio-7cc01a2153ef487c355ed3b8754b498545e534937c4cda757292df0e434f6a48 WatchSource:0}: Error finding container 7cc01a2153ef487c355ed3b8754b498545e534937c4cda757292df0e434f6a48: Status 404 returned error can't find the container with id 7cc01a2153ef487c355ed3b8754b498545e534937c4cda757292df0e434f6a48 Nov 03 20:51:45.388398 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.388360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:45.903671 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.903630 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" event={"ID":"4492c45a-e540-4512-9ee9-f137628c4ed3","Type":"ContainerStarted","Data":"8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616"} Nov 03 20:51:45.904125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.903695 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" event={"ID":"4492c45a-e540-4512-9ee9-f137628c4ed3","Type":"ContainerStarted","Data":"7cc01a2153ef487c355ed3b8754b498545e534937c4cda757292df0e434f6a48"} Nov 03 20:51:45.904125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.903821 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:45.905138 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.905111 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" event={"ID":"2a9c378e-f855-4925-af28-a2abc3124493","Type":"ContainerStarted","Data":"0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4"} Nov 03 20:51:45.905285 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.905234 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:45.954311 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.954270 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" podStartSLOduration=1.954258797 podStartE2EDuration="1.954258797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:45.941002008 +0000 UTC m=+2887.407957274" watchObservedRunningTime="2026-11-03 20:51:45.954258797 +0000 UTC m=+2887.421214050" Nov 03 20:51:45.980202 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:45.980161 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" podStartSLOduration=2.622798261 podStartE2EDuration="2.980149441s" totalImagesPullingTime="357.35118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:45.979122246 +0000 UTC m=+2887.446077499" watchObservedRunningTime="2026-11-03 20:51:45.980149441 +0000 UTC m=+2887.447104695" Nov 03 20:51:47.118190 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:47.118151 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:48.803752 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:48.803715 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:50.614693 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:50.614656 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:51.916588 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:51.916553 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:51:52.401413 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:52.401376 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:54.100542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:54.100509 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:55.777718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:55.777677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:56.914252 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:56.914222 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:51:57.615509 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:57.615472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:58.467973 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:58.467939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.434064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:51:59.434027 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:01.214925 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:01.214886 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:03.007986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:03.007947 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:04.768432 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:04.768402 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:05.627317 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:52:05.627283 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:52:06.372431 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:06.372395 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:07.990438 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:07.990405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:08.137173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:08.137141 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm"] Nov 03 20:52:08.285688 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:08.285595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:08.387521 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:08.387484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:09.684195 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:09.684156 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:11.066508 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.066473 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn"] Nov 03 20:52:11.066988 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.066714 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" podUID="2a9c378e-f855-4925-af28-a2abc3124493" containerName="maas-api" containerID="cri-o://0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4" gracePeriod=30 Nov 03 20:52:11.128537 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.128502 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:11.343516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.343490 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:52:11.441442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.441407 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca\") pod \"2a9c378e-f855-4925-af28-a2abc3124493\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " Nov 03 20:52:11.441634 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.441608 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf\" (UniqueName: \"kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf\") pod \"2a9c378e-f855-4925-af28-a2abc3124493\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " Nov 03 20:52:11.441713 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.441643 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\") pod \"2a9c378e-f855-4925-af28-a2abc3124493\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " Nov 03 20:52:11.441776 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.441745 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs\") pod \"2a9c378e-f855-4925-af28-a2abc3124493\" (UID: \"2a9c378e-f855-4925-af28-a2abc3124493\") " Nov 03 20:52:11.441869 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.441829 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca" pod "2a9c378e-f855-4925-af28-a2abc3124493" (UID: "2a9c378e-f855-4925-af28-a2abc3124493"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:11.442161 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.442134 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9c378e-f855-4925-af28-a2abc3124493-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:52:11.443821 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.443794 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls" pod "2a9c378e-f855-4925-af28-a2abc3124493" (UID: "2a9c378e-f855-4925-af28-a2abc3124493"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:11.443821 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.443807 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf" pod "2a9c378e-f855-4925-af28-a2abc3124493" (UID: "2a9c378e-f855-4925-af28-a2abc3124493"). InnerVolumeSpecName "kube-api-access-6f4hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:11.443966 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.443826 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs" pod "2a9c378e-f855-4925-af28-a2abc3124493" (UID: "2a9c378e-f855-4925-af28-a2abc3124493"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:11.543021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.542985 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:52:11.543021 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.543024 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f4hf\" (UniqueName: \"kubernetes.io/projected/2a9c378e-f855-4925-af28-a2abc3124493-kube-api-access-6f4hf\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:52:11.543257 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.543039 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9c378e-f855-4925-af28-a2abc3124493-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:52:11.613476 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.613427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:11.714422 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:11.714384 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:12.061010 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.060922 5043 generic.go:356] "Generic (PLEG): container finished" podID="2a9c378e-f855-4925-af28-a2abc3124493" containerID="0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4" exitCode=0 Nov 03 20:52:12.061010 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.060970 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" Nov 03 20:52:12.061242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.061015 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" event={"ID":"2a9c378e-f855-4925-af28-a2abc3124493","Type":"ContainerDied","Data":"0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4"} Nov 03 20:52:12.061242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.061068 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn" event={"ID":"2a9c378e-f855-4925-af28-a2abc3124493","Type":"ContainerDied","Data":"09c470852a2397e181eb47c79c4f315a2d48c44d9a42117b688270d50271d5a6"} Nov 03 20:52:12.061242 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.061127 5043 scope.go:122] "RemoveContainer" containerID="0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4" Nov 03 20:52:12.073026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.072822 5043 scope.go:122] "RemoveContainer" containerID="0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4" Nov 03 20:52:12.073306 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:52:12.073132 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4\": container with ID starting with 0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4 not found: ID does not exist" containerID="0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4" Nov 03 20:52:12.073306 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.073161 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4"} err="failed to get container status \"0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4\": rpc error: code = NotFound desc = could not find container \"0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4\": container with ID starting with 0c32c6f68e59e47fe039a8b3e1069dc56a8eb2e060b0ae496289ef24720481d4 not found: ID does not exist" Nov 03 20:52:12.114353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.114326 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn"] Nov 03 20:52:12.126083 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.126044 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2966c90-67766f4c47-pqdvn"] Nov 03 20:52:12.676568 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:12.676535 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9c378e-f855-4925-af28-a2abc3124493" path="/var/lib/kubelet/pods/2a9c378e-f855-4925-af28-a2abc3124493/volumes" Nov 03 20:52:13.137736 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.137689 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" podUID="4492c45a-e540-4512-9ee9-f137628c4ed3" containerName="payload-processing" containerID="cri-o://8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616" gracePeriod=25 Nov 03 20:52:13.400353 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.400281 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:52:13.563960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.563921 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn\" (UniqueName: \"kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn\") pod \"4492c45a-e540-4512-9ee9-f137628c4ed3\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " Nov 03 20:52:13.564163 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.564111 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume\") pod \"4492c45a-e540-4512-9ee9-f137628c4ed3\" (UID: \"4492c45a-e540-4512-9ee9-f137628c4ed3\") " Nov 03 20:52:13.564444 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.564410 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume" pod "4492c45a-e540-4512-9ee9-f137628c4ed3" (UID: "4492c45a-e540-4512-9ee9-f137628c4ed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:13.564593 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.564535 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4492c45a-e540-4512-9ee9-f137628c4ed3-plugins-config-volume\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:52:13.566153 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.566128 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn" pod "4492c45a-e540-4512-9ee9-f137628c4ed3" (UID: "4492c45a-e540-4512-9ee9-f137628c4ed3"). InnerVolumeSpecName "kube-api-access-n89dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:13.665420 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:13.665342 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89dn\" (UniqueName: \"kubernetes.io/projected/4492c45a-e540-4512-9ee9-f137628c4ed3-kube-api-access-n89dn\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:52:14.022986 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.022898 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:14.074569 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.074537 5043 generic.go:356] "Generic (PLEG): container finished" podID="4492c45a-e540-4512-9ee9-f137628c4ed3" containerID="8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616" exitCode=0 Nov 03 20:52:14.074734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.074586 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" Nov 03 20:52:14.074734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.074623 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" event={"ID":"4492c45a-e540-4512-9ee9-f137628c4ed3","Type":"ContainerDied","Data":"8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616"} Nov 03 20:52:14.074734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.074672 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm" event={"ID":"4492c45a-e540-4512-9ee9-f137628c4ed3","Type":"ContainerDied","Data":"7cc01a2153ef487c355ed3b8754b498545e534937c4cda757292df0e434f6a48"} Nov 03 20:52:14.074734 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.074697 5043 scope.go:122] "RemoveContainer" containerID="8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616" Nov 03 20:52:14.088336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.088309 5043 scope.go:122] "RemoveContainer" containerID="8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616" Nov 03 20:52:14.089254 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:52:14.089226 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616\": container with ID starting with 8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616 not found: ID does not exist" containerID="8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616" Nov 03 20:52:14.089356 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.089265 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616"} err="failed to get container status \"8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616\": rpc error: code = NotFound desc = could not find container \"8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616\": container with ID starting with 8283e21ceb85ec869e7ec44f70b3a0bdc15dd3d82e8d2770ec74d12014567616 not found: ID does not exist" Nov 03 20:52:14.146617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.146583 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm"] Nov 03 20:52:14.158453 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.158427 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2966c90-656cdd4bfd-hqzjm"] Nov 03 20:52:14.676773 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:14.676742 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4492c45a-e540-4512-9ee9-f137628c4ed3" path="/var/lib/kubelet/pods/4492c45a-e540-4512-9ee9-f137628c4ed3/volumes" Nov 03 20:52:15.271559 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:15.271521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:15.730916 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:15.730868 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:15.882410 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:15.882372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.736782 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:18.736751 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.838272 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:18.838232 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.153580 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.153550 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n"] Nov 03 20:52:20.154029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.154016 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4492c45a-e540-4512-9ee9-f137628c4ed3" containerName="payload-processing" Nov 03 20:52:20.154102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.154030 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4492c45a-e540-4512-9ee9-f137628c4ed3" containerName="payload-processing" Nov 03 20:52:20.154102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.154063 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2a9c378e-f855-4925-af28-a2abc3124493" containerName="maas-api" Nov 03 20:52:20.154102 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.154067 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9c378e-f855-4925-af28-a2abc3124493" containerName="maas-api" Nov 03 20:52:20.154214 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.154156 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4492c45a-e540-4512-9ee9-f137628c4ed3" containerName="payload-processing" Nov 03 20:52:20.154214 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.154169 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9c378e-f855-4925-af28-a2abc3124493" containerName="maas-api" Nov 03 20:52:20.158567 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.158549 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.174913 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.174898 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-d04674-serving-cert\"" Nov 03 20:52:20.190542 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.190513 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n"] Nov 03 20:52:20.248050 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.248012 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.332265 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.332234 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.332415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.332287 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.332488 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.332469 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68tg8\" (UniqueName: \"kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.332536 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.332511 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.348146 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.348119 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.433799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.433721 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tg8\" (UniqueName: \"kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.433799 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.433778 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.434042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.433826 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.434042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.433857 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.434644 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.434619 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.436433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.436412 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.436540 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.436463 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.452442 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.452408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.456429 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.456409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tg8\" (UniqueName: \"kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8\") pod \"maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.472217 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.472191 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:20.556037 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.555999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.773935 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:20.773913 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n"] Nov 03 20:52:20.776541 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:52:20.776515 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc243ee6_d504_4b3c_b97e_cbaaabc8516f.slice/crio-4cae0daff3077b2f84928c254bd6f4f7491c233254d4ca606e73c94fe339020e WatchSource:0}: Error finding container 4cae0daff3077b2f84928c254bd6f4f7491c233254d4ca606e73c94fe339020e: Status 404 returned error can't find the container with id 4cae0daff3077b2f84928c254bd6f4f7491c233254d4ca606e73c94fe339020e Nov 03 20:52:21.117663 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:21.117629 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" event={"ID":"fc243ee6-d504-4b3c-b97e-cbaaabc8516f","Type":"ContainerStarted","Data":"4cae0daff3077b2f84928c254bd6f4f7491c233254d4ca606e73c94fe339020e"} Nov 03 20:52:21.982665 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:21.982615 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.127768 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:22.127727 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" event={"ID":"fc243ee6-d504-4b3c-b97e-cbaaabc8516f","Type":"ContainerStarted","Data":"05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f"} Nov 03 20:52:22.127945 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:22.127802 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:22.170996 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:22.170935 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" podStartSLOduration=1.805184565 podStartE2EDuration="2.170917603s" totalImagesPullingTime="365.733038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:22.168346458 +0000 UTC m=+2923.635301705" watchObservedRunningTime="2026-11-03 20:52:22.170917603 +0000 UTC m=+2923.637872858" Nov 03 20:52:23.670067 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:23.670034 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:25.406900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:25.406864 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:27.197904 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:27.197867 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:28.139656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:28.139626 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:52:28.888369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:28.888331 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:30.680040 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:30.679994 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:32.292362 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:32.292323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:33.617422 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:33.617384 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:34.013812 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:34.013718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:34.229728 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:34.229686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:35.628037 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:52:35.628001 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:52:35.824570 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:35.824541 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:37.594951 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:37.594919 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:39.401957 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:39.401918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:41.071619 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:41.071581 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:42.798173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:42.798140 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:44.394891 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:44.394842 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:45.969321 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:45.969279 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:47.671551 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:47.671510 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:49.414289 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:49.414253 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:50.989471 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:50.989424 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:52.565347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:52.565313 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:54.169125 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:54.169088 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:55.857855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:55.857820 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:57.567705 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:57.567670 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:59.168936 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:52:59.168899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:00.784709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:00.784677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:02.469183 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:02.469153 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:04.063854 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:04.063821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:05.627933 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:53:05.627894 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:53:05.905052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:05.904969 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:07.580662 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:07.580630 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:09.225636 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:09.225592 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:10.872600 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:10.872568 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:12.483149 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:12.483065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:14.172822 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:14.172786 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.772564 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:15.772530 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:17.379212 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:17.379179 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:19.077864 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:19.077826 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:20.830977 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:20.830935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:22.369802 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:22.369764 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:24.075724 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:24.075685 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:25.662343 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:25.662309 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:27.368178 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:27.368133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:28.995036 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:28.994989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:30.694185 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:30.694148 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:32.477863 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:32.477828 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:34.170270 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:34.170233 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:34.339901 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:34.339866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:35.628148 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:53:35.628063 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:53:35.771304 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:35.771269 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:37.473372 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:37.473339 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:39.095779 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:39.095744 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:39.335049 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:39.335018 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:53:39.340888 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:39.340866 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:53:39.341804 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:39.341776 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:53:39.347918 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:39.347871 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2chqk_331068c4-2a21-4ab6-9fbf-ea1ce605d767/ovn-acl-logging/0.log" Nov 03 20:53:40.805958 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:40.805922 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:42.369418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:42.369376 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:43.970887 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:43.970855 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:45.569725 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:45.569683 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:47.269602 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:47.269566 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:48.872960 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:48.872921 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:50.485096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:50.485031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:52.169158 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:52.169048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:53.888047 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:53.888005 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:55.582505 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:55.582467 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:57.242243 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:57.242212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:58.907715 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:53:58.907682 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:00.569770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:00.569730 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:02.174315 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:02.174274 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:03.808470 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:03.808430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:04.117268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:04.117235 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:04.217186 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:04.217151 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:05.597656 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:05.597615 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:05.628725 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:54:05.628692 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:54:12.095622 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.095553 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n"] Nov 03 20:54:12.096590 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.096552 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" podUID="fc243ee6-d504-4b3c-b97e-cbaaabc8516f" containerName="maas-api" containerID="cri-o://05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f" gracePeriod=30 Nov 03 20:54:12.153706 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.153669 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:12.378939 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.378916 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:54:12.519746 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.519711 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls\") pod \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " Nov 03 20:54:12.519900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.519803 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs\") pod \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " Nov 03 20:54:12.519900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.519867 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca\") pod \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " Nov 03 20:54:12.519900 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.519895 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8\" (UniqueName: \"kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8\") pod \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\" (UID: \"fc243ee6-d504-4b3c-b97e-cbaaabc8516f\") " Nov 03 20:54:12.520322 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.520292 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca" pod "fc243ee6-d504-4b3c-b97e-cbaaabc8516f" (UID: "fc243ee6-d504-4b3c-b97e-cbaaabc8516f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:12.522029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.522000 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls" pod "fc243ee6-d504-4b3c-b97e-cbaaabc8516f" (UID: "fc243ee6-d504-4b3c-b97e-cbaaabc8516f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:12.522173 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.522090 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8" pod "fc243ee6-d504-4b3c-b97e-cbaaabc8516f" (UID: "fc243ee6-d504-4b3c-b97e-cbaaabc8516f"). InnerVolumeSpecName "kube-api-access-68tg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:12.522293 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.522272 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs" pod "fc243ee6-d504-4b3c-b97e-cbaaabc8516f" (UID: "fc243ee6-d504-4b3c-b97e-cbaaabc8516f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:12.621673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.621590 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-openshift-service-ca\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:54:12.621673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.621618 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68tg8\" (UniqueName: \"kubernetes.io/projected/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-kube-api-access-68tg8\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:54:12.621673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.621628 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-maas-api-tls\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:54:12.621673 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.621636 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc243ee6-d504-4b3c-b97e-cbaaabc8516f-metrics-certs\") on node \"ip-10-0-140-102.ec2.internal\" DevicePath \"\"" Nov 03 20:54:12.666523 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.666482 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:12.739718 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.739684 5043 generic.go:356] "Generic (PLEG): container finished" podID="fc243ee6-d504-4b3c-b97e-cbaaabc8516f" containerID="05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f" exitCode=0 Nov 03 20:54:12.739902 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.739733 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" Nov 03 20:54:12.739902 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.739769 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" event={"ID":"fc243ee6-d504-4b3c-b97e-cbaaabc8516f","Type":"ContainerDied","Data":"05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f"} Nov 03 20:54:12.739902 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.739825 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n" event={"ID":"fc243ee6-d504-4b3c-b97e-cbaaabc8516f","Type":"ContainerDied","Data":"4cae0daff3077b2f84928c254bd6f4f7491c233254d4ca606e73c94fe339020e"} Nov 03 20:54:12.739902 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.739853 5043 scope.go:122] "RemoveContainer" containerID="05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f" Nov 03 20:54:12.748952 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.748923 5043 scope.go:122] "RemoveContainer" containerID="05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f" Nov 03 20:54:12.749279 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:54:12.749256 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f\": container with ID starting with 05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f not found: ID does not exist" containerID="05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f" Nov 03 20:54:12.749371 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.749287 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f"} err="failed to get container status \"05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f\": rpc error: code = NotFound desc = could not find container \"05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f\": container with ID starting with 05aaf3d500df491d6fdfae3b66637c652084d81ebaa40c8b2cd53697734a848f not found: ID does not exist" Nov 03 20:54:12.761667 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.761645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:12.790026 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.789966 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n"] Nov 03 20:54:12.802170 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:12.802137 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d04674-56f784d849-ldb9n"] Nov 03 20:54:13.358791 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:13.358755 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:14.676801 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:14.676762 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc243ee6-d504-4b3c-b97e-cbaaabc8516f" path="/var/lib/kubelet/pods/fc243ee6-d504-4b3c-b97e-cbaaabc8516f/volumes" Nov 03 20:54:19.096032 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:19.095993 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:20.592418 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:20.592385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:34.468814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:34.468781 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:35.627553 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:54:35.627512 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:54:53.110096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:53.109165 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmjj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:53.110096 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:54:53.109260 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmjj5" podUID="02ccc348-aabb-4a1d-a3d5-f8826969be95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:55:05.633588 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:55:05.633540 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:55:26.826885 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:26.826814 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kl5jp_129a77fa-fdc8-4ab0-b177-6a798ca33943/manager/0.log" Nov 03 20:55:27.036971 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:27.036937 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p4zcv_8d7494ea-7a36-4407-9375-92ae8247bc26/manager/0.log" Nov 03 20:55:27.811415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:27.811385 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4mwwp_56690b38-ee17-4993-9dde-eb4e499dd9b8/server/0.log" Nov 03 20:55:28.003687 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:28.003653 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-b474r_123dcd1e-fd84-4fff-b88d-0615c9d12cd9/manager/0.log" Nov 03 20:55:28.227056 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:28.227020 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rl7rp_5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd/proxy/0.log" Nov 03 20:55:28.954738 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:28.954705 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796c56b65d-pzs9h_344c8e8e-1434-4e8e-be26-bed9f902dca1/maas-api/0.log" Nov 03 20:55:30.459828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:30.459796 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-mbrgl_57422d5e-d525-4e5c-bed1-f0b3258f9c46/postgres/0.log" Nov 03 20:55:31.150794 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:31.150760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.889351 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:31.889284 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.989052 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:31.988995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:32.140159 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:32.140066 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kg5pr_5f971215-b5d6-469e-8374-b2beeba39c28/manager/0.log" Nov 03 20:55:32.347253 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:32.347227 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kqx9m_f84e9b68-be0d-434c-8a18-d3139cc01b6d/manager/0.log" Nov 03 20:55:32.761245 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:32.761219 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4jslm_715fba24-96d9-4967-a526-25859e8ea8e1/manager/0.log" Nov 03 20:55:33.951475 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:33.951434 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr_23b2d99d-1f21-431b-9e74-8e16005ede32/istio-proxy/0.log" Nov 03 20:55:34.572223 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:34.572186 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:34.735631 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:34.735602 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-5mbh8_2044f915-8f34-4c29-ac6b-754910c41df8/istio-proxy/0.log" Nov 03 20:55:35.286833 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:35.286805 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dl6vz_17893f17-0fe4-4378-b6cd-f2ee0dfebd26/istio-proxy/1.log" Nov 03 20:55:35.627693 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:55:35.627657 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:55:36.305682 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:36.305650 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-rmjj5_02ccc348-aabb-4a1d-a3d5-f8826969be95/istio-proxy/0.log" Nov 03 20:55:36.691513 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:36.691483 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr_30b84b2e-4811-4823-824d-63308ffb9739/istio-proxy/0.log" Nov 03 20:55:36.877347 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:36.877322 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tdvk7_6077b5c3-e90d-46e9-88c7-07020c0c8e83/payload-pre-processing/0.log" Nov 03 20:55:37.042045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:37.041948 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:55:37.227064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:37.227030 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d68796f8f-nhd5b_c2062bc3-f9db-48f9-aedb-b1367774ee26/router/0.log" Nov 03 20:55:37.676045 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:37.676014 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dl6vz_17893f17-0fe4-4378-b6cd-f2ee0dfebd26/istio-proxy/0.log" Nov 03 20:55:38.164055 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:38.164029 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m_a324baa7-3f3d-4bb1-b80e-185ac8be80ee/main/0.log" Nov 03 20:55:38.535136 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:38.535029 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fcnhk_39e57714-521a-402a-b401-36d07bdd2b3b/main/0.log" Nov 03 20:55:38.734346 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:38.734321 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4q9mv_45b70aee-391e-426c-b61b-c49fd5f58c73/main/0.log" Nov 03 20:55:39.117563 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:39.117536 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qhg57_e77bfbd0-7444-45e1-871b-cafaa9b9f1b1/main/0.log" Nov 03 20:55:48.953174 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:48.953141 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kl5jp_129a77fa-fdc8-4ab0-b177-6a798ca33943/manager/0.log" Nov 03 20:55:49.158230 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:49.158196 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-p4zcv_8d7494ea-7a36-4407-9375-92ae8247bc26/manager/0.log" Nov 03 20:55:49.838911 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:49.838881 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4mwwp_56690b38-ee17-4993-9dde-eb4e499dd9b8/server/0.log" Nov 03 20:55:50.017574 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:50.017544 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-b474r_123dcd1e-fd84-4fff-b88d-0615c9d12cd9/manager/0.log" Nov 03 20:55:50.231268 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:50.231186 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rl7rp_5b6a6937-5f28-4ca9-857c-7a3ec8d1aadd/proxy/0.log" Nov 03 20:55:50.745088 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:50.745047 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796c56b65d-pzs9h_344c8e8e-1434-4e8e-be26-bed9f902dca1/maas-api/0.log" Nov 03 20:55:52.029709 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:52.029674 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-mbrgl_57422d5e-d525-4e5c-bed1-f0b3258f9c46/postgres/0.log" Nov 03 20:55:53.348165 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:53.348136 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kg5pr_5f971215-b5d6-469e-8374-b2beeba39c28/manager/0.log" Nov 03 20:55:53.553632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:53.553599 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kqx9m_f84e9b68-be0d-434c-8a18-d3139cc01b6d/manager/0.log" Nov 03 20:55:53.929556 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:53.929528 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4jslm_715fba24-96d9-4967-a526-25859e8ea8e1/manager/0.log" Nov 03 20:55:54.893002 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:54.892970 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr_23b2d99d-1f21-431b-9e74-8e16005ede32/istio-proxy/0.log" Nov 03 20:55:55.635172 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:55.635143 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-5mbh8_2044f915-8f34-4c29-ac6b-754910c41df8/istio-proxy/0.log" Nov 03 20:55:56.223029 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:56.222999 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dl6vz_17893f17-0fe4-4378-b6cd-f2ee0dfebd26/istio-proxy/1.log" Nov 03 20:55:57.040813 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:57.040780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:57.197737 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:57.197710 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-rmjj5_02ccc348-aabb-4a1d-a3d5-f8826969be95/istio-proxy/0.log" Nov 03 20:55:57.573384 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:57.573354 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr_30b84b2e-4811-4823-824d-63308ffb9739/istio-proxy/0.log" Nov 03 20:55:57.758064 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:57.758035 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tdvk7_6077b5c3-e90d-46e9-88c7-07020c0c8e83/payload-pre-processing/0.log" Nov 03 20:55:57.925632 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:57.925611 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:55:58.110862 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:58.110832 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d68796f8f-nhd5b_c2062bc3-f9db-48f9-aedb-b1367774ee26/router/0.log" Nov 03 20:55:58.637433 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:58.637406 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xdr2m_a324baa7-3f3d-4bb1-b80e-185ac8be80ee/main/0.log" Nov 03 20:55:59.027421 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:59.027338 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fcnhk_39e57714-521a-402a-b401-36d07bdd2b3b/main/0.log" Nov 03 20:55:59.216280 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:59.216251 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4q9mv_45b70aee-391e-426c-b61b-c49fd5f58c73/main/0.log" Nov 03 20:55:59.567786 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:55:59.567756 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qhg57_e77bfbd0-7444-45e1-871b-cafaa9b9f1b1/main/0.log" Nov 03 20:56:04.492156 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:04.492119 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:04.585415 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:04.585388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:05.628209 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:56:05.628170 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:56:30.799400 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:30.799367 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:56:30.972669 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:30.972637 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:56:34.701245 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:34.701202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:35.627996 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:56:35.627955 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:56:43.832326 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:43.832278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:59.152769 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:59.152683 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:59.239369 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:56:59.239329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:05.627567 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:57:05.627525 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:57:09.560237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:09.560207 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:57:09.754605 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:09.754574 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:57:10.073938 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:10.073908 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:10.195367 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:10.195327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:14.538719 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:14.538687 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kxtqd_25244768-eb42-47ea-a28d-20609fc12f42/global-pull-secret-syncer/0.log" Nov 03 20:57:14.709962 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:14.709932 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-htm8c_47ee139e-2eb1-4e93-9536-7c36c003ff40/connection-checker/0.log" Nov 03 20:57:14.829516 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:14.829492 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tc29n_aa34540b-36c1-4ef4-b4c9-e14a2419ee31/connection-checker/0.log" Nov 03 20:57:14.949254 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:14.949230 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wh79h_dfd74d5b-c18f-42cb-be6d-416d2eb1c921/connection-checker/0.log" Nov 03 20:57:15.106845 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:15.106766 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" Nov 03 20:57:15.820655 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:15.820626 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-102.ec2.internal_880c789d3d697e88c74ab8f8cf26d9d3/haproxy/0.log" Nov 03 20:57:21.421349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:21.421313 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kg5pr_5f971215-b5d6-469e-8374-b2beeba39c28/manager/0.log" Nov 03 20:57:21.542847 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:21.542819 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kqx9m_f84e9b68-be0d-434c-8a18-d3139cc01b6d/manager/0.log" Nov 03 20:57:21.818091 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:21.817996 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4jslm_715fba24-96d9-4967-a526-25859e8ea8e1/manager/0.log" Nov 03 20:57:23.625461 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.625428 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm"] Nov 03 20:57:23.626024 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.626006 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fc243ee6-d504-4b3c-b97e-cbaaabc8516f" containerName="maas-api" Nov 03 20:57:23.626090 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.626025 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc243ee6-d504-4b3c-b97e-cbaaabc8516f" containerName="maas-api" Nov 03 20:57:23.626145 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.626135 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc243ee6-d504-4b3c-b97e-cbaaabc8516f" containerName="maas-api" Nov 03 20:57:23.629617 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.629599 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.644390 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.644346 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fng2b\"/\"default-dockercfg-6f4g7\"" Nov 03 20:57:23.647349 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.647325 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm"] Nov 03 20:57:23.668941 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.668909 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-proc\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.669139 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.669114 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-lib-modules\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.669225 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.669176 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-sys\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.669283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.669221 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-podres\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.669283 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.669269 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqz5g\" (UniqueName: \"kubernetes.io/projected/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-kube-api-access-mqz5g\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.734042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.734006 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fng2b\"/\"openshift-service-ca.crt\"" Nov 03 20:57:23.734042 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.734024 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fng2b\"/\"kube-root-ca.crt\"" Nov 03 20:57:23.770577 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770542 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-proc\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770636 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-lib-modules\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770678 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-sys\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770689 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-proc\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770711 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-podres\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770756 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqz5g\" (UniqueName: \"kubernetes.io/projected/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-kube-api-access-mqz5g\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770774 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-podres\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770781 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-sys\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.770828 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.770781 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-lib-modules\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.789999 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.789972 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqz5g\" (UniqueName: \"kubernetes.io/projected/13da9fb2-d08c-41fc-bfbb-13e3b30b1055-kube-api-access-mqz5g\") pod \"perf-node-gather-daemonset-pwpkm\" (UID: \"13da9fb2-d08c-41fc-bfbb-13e3b30b1055\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:23.942550 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:23.942468 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:24.236949 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.236925 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm"] Nov 03 20:57:24.239449 ip-10-0-140-102 kubenswrapper[5043]: W1103 20:57:24.239416 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13da9fb2_d08c_41fc_bfbb_13e3b30b1055.slice/crio-41d2cb9910c0411a9c6680a43656788a5293a26977d68a3fea6d544bf35d7365 WatchSource:0}: Error finding container 41d2cb9910c0411a9c6680a43656788a5293a26977d68a3fea6d544bf35d7365: Status 404 returned error can't find the container with id 41d2cb9910c0411a9c6680a43656788a5293a26977d68a3fea6d544bf35d7365 Nov 03 20:57:24.302621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.302601 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-lv272_e05b9d70-936b-44f5-b0f2-bc68492d7951/cluster-monitoring-operator/0.log" Nov 03 20:57:24.872814 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.872774 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" event={"ID":"13da9fb2-d08c-41fc-bfbb-13e3b30b1055","Type":"ContainerStarted","Data":"55da759df1fc2fbb2d5f542c73ab32383bc1c4c7ff183b89fdca6ed599611365"} Nov 03 20:57:24.873320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.872838 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" event={"ID":"13da9fb2-d08c-41fc-bfbb-13e3b30b1055","Type":"ContainerStarted","Data":"41d2cb9910c0411a9c6680a43656788a5293a26977d68a3fea6d544bf35d7365"} Nov 03 20:57:24.873320 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.872917 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:24.909917 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.909867 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" podStartSLOduration=1.909852598 podStartE2EDuration="1.909852598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:24.909577274 +0000 UTC m=+3226.376532527" watchObservedRunningTime="2026-11-03 20:57:24.909852598 +0000 UTC m=+3226.376807853" Nov 03 20:57:24.983798 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:24.983770 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-87rfg_ee70c47d-21c7-433d-9d08-87df444b9235/node-exporter/0.log" Nov 03 20:57:25.094855 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:25.094831 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-87rfg_ee70c47d-21c7-433d-9d08-87df444b9235/kube-rbac-proxy/0.log" Nov 03 20:57:25.206336 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:25.206258 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-87rfg_ee70c47d-21c7-433d-9d08-87df444b9235/init-textfile/0.log" Nov 03 20:57:27.437621 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:27.437593 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-x2m62_1161d01c-a20b-4036-be74-22978c149f7d/prometheus-operator-admission-webhook/0.log" Nov 03 20:57:29.882770 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:29.882743 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-9mgnl_85530c5c-c1ec-4d44-8a76-1ea11f34c7a2/networking-console-plugin/0.log" Nov 03 20:57:30.884083 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:30.884030 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/2.log" Nov 03 20:57:30.890079 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:30.890052 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-pwpkm" Nov 03 20:57:30.891545 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:30.891521 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6l254_9a28d511-b41b-4b0c-8819-6cba4a6620d8/console-operator/3.log" Nov 03 20:57:31.912333 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:31.912298 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-5phwq_2ae1a8b7-4dd4-4e7a-8f29-c08ec80f10ef/download-server/0.log" Nov 03 20:57:33.030535 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:33.030499 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-kvjms_a22c4edd-da5c-44d4-a71c-6e162cf7f221/volume-data-source-validator/0.log" Nov 03 20:57:34.807668 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:34.807623 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bsd7_0a8e9079-e494-4815-a2dc-fc3aa2763628/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:35.508777 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:35.508746 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr_23b2d99d-1f21-431b-9e74-8e16005ede32/istio-proxy/0.log" Nov 03 20:57:35.608511 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:35.608474 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5rssm_a7ff3150-0ab6-42d2-81ab-b03e7d944674/dns/0.log" Nov 03 20:57:35.628584 ip-10-0-140-102 kubenswrapper[5043]: E1103 20:57:35.628550 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nkf6" containerName="extractor" Nov 03 20:57:35.754890 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:35.754863 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5rssm_a7ff3150-0ab6-42d2-81ab-b03e7d944674/kube-rbac-proxy/0.log" Nov 03 20:57:36.359818 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:36.359791 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zmtdq_26f26e13-d1c9-4747-914f-af2fb15d7ed7/dns-node-resolver/0.log" Nov 03 20:57:36.768806 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:36.768722 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-5mbh8_2044f915-8f34-4c29-ac6b-754910c41df8/istio-proxy/0.log" Nov 03 20:57:36.957576 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:36.957544 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dl6vz_17893f17-0fe4-4378-b6cd-f2ee0dfebd26/istio-proxy/0.log" Nov 03 20:57:36.959778 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:36.959752 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dl6vz_17893f17-0fe4-4378-b6cd-f2ee0dfebd26/istio-proxy/1.log" Nov 03 20:57:37.270802 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.270774 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-rmjj5_02ccc348-aabb-4a1d-a3d5-f8826969be95/istio-proxy/0.log" Nov 03 20:57:37.394583 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.394551 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-wxqpr_30b84b2e-4811-4823-824d-63308ffb9739/istio-proxy/0.log" Nov 03 20:57:37.423277 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.423250 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-dc8df666f-5wrpc_6c7fb94d-1281-4d31-8413-b97a6c0389f8/registry/0.log" Nov 03 20:57:37.456684 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.456657 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tdvk7_6077b5c3-e90d-46e9-88c7-07020c0c8e83/payload-pre-processing/0.log" Nov 03 20:57:37.516854 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.516824 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-562kq_463353e1-4394-41c6-8dae-59ff6f3208c1/payload-processing/0.log" Nov 03 20:57:37.521060 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.520993 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bkb5w_fff6f118-0ff9-4478-b759-04e2d3f63420/node-ca/0.log" Nov 03 20:57:37.589664 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:37.589641 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d68796f8f-nhd5b_c2062bc3-f9db-48f9-aedb-b1367774ee26/router/0.log" Nov 03 20:57:39.480028 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:39.479989 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr_23b2d99d-1f21-431b-9e74-8e16005ede32/istio-proxy/0.log" Nov 03 20:57:40.785237 ip-10-0-140-102 kubenswrapper[5043]: I1103 20:57:40.785206 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-tgzhr_23b2d99d-1f21-431b-9e74-8e16005ede32/istio-proxy/0.log"